Add --pod-pending-timeout config option

[jeremybumsted]

Currently, the controller has a handful of different levers to control when a pod will get terminated, specifically when a pod has been stuck in Pending for a long time:

• EmptyJobGracePeriod is set for a job that has started but never created a pod
• ImagePullBackOffGracePeriod is for when the pods are created but stuck in the ImagePullBackOff state
• JobActiveDeadlineSeconds would handle pretty much every other case, like those pods that are generally pending due to insufficient cluster resources, mismatched node selectors, and so on.

This means that the closest thing to handling pods stuck in the Pending state is JobActiveDeadlineSeconds, which accounts for every stage of the job lifecycle on the Buildkite side (including pods that successfully start and are running a job). It also has a default of 6 hours - ample time for a running job, but if you've got pods stuck and you want them to fail faster, you'll be stuck with pending pods for quite a while.

This PR aims to introduce a PodPendingTimeout configuration option, which allows the controller to be configured with a timeout for when a pod has been Pending without start the job for the configured amount of time, default 15 minutes.

Potential Caveat: This does introduce a new goroutine to the podWatcher in order to handle watching the pending state, I don't know if that's an ideal way to do this, so very open to suggestions on better possible ways to do this.

Includes tests, including integration tests to validate the timeouts.

I've also tested this locally by configuring a pod-pending-timeout to 30s:

config:
    debug: true
    pod-pending-timeout: 30s
    queue: kubernetes

and can confirm successful job failure due to the pending timeout 👇

❓some open questions

• I landed on a default timeout period of 15 minutes; that feels like a reasonable amount of time for a pod to start, but perhaps it would be better to be an optional config (no config value set = no pod pending timeout)?
• Does this feel like I'm introducing a sharp edge into the controller? (maybe removing a default timeout would be better?)

[DrJosh9000]

This is very good! I have some stylistic comments.

[jeremybumsted]

Thank you @DrJosh9000! Applied the suggestions and learned some things 💪

Also removed a test case that is no longer needed due to the change removing the explicit check for createdAt.IsZero().

[DrJosh9000]

Good work!