PB-133: feature flag for job reservation support

[zhming0]

This feature flag currently does nothing, but it will de-risk our incoming job reservation related changes to k8s stack.

Also, hopefully this gently conveys a message that we are doubling down our investment in our k8s stack offering 🙂

[DrJosh9000]

I have a can of worms over the flag description, but LGTM

[DrJosh9000]

1. Contact us how? Email to support, file an issue?
2. Contact us why? Currently the flag does nothing, and in future when it does do something, it's marked as "experimental". It's better for us if users want to try it, and "contact us before enabling" is a roadblock to that. But we definitely want to know if they try it and hit a bug.

I'm not sure how best to phrase it, but:

Suggested change

	"title": "Please contact us before enabling this flag. This feature enables job reservation support: Better job observability, scalable job fetching!",
	"title": "Experimental - may contain bugs or cause breakage. This experiment enables job reservation support for better job observability and scalable job fetching. If you try it, please let us know about your experiences by filing an issue on https://github.com/buildkite/agent-stack-k8s",

[moskyb]

i suspect the initial motivation for "contact us before using" is on the proviso that using the job reservation endpoint will be feature flagged on the backend, but... now that i'm thinking about it, it's probably worth thinking about not feature-flagging the endpoint (on the backend). any use of it will be opt-in, either by using this flag, or by just choosing to call the endpoint, so all of the caveats have more or less been emptor'd at that stage.

feature flags are useful for backend stuff where the user doesn't have much choice over what's presented to them, but the utility of a feature flag is diminished when a user has to actively choose to use something anyway -- it's just more busywork for us turning it on and off for people.

[zhming0]

@moskyb yeah, I am not very worried about the backend feature flag either. I just am conscious that there will be a period of time when calling "reserved" api means a dead job, so I was thinking if we should de-risk -- only allow such invocation internally. But it's quite a small risk for sure.

@DrJosh9000 I like your version, with a small modification: Experimental - does not fully function yet.. I think at this stage of the project, we want to be clear that this is not working. We can later come back and change that when we are almost ready.

[DrJosh9000]

Ah... I had meant to remind you to update this text to be consistent with the JSON schema.

[zhming0]

Oops quick follow up: #647