Kubernetes: allow opt-out of rescheduling on SIGTERM by setting PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR

[svatopluk-sperka]

closes #18867.

Gist of the problem is that currently there is no way to opt-out of rescheduling a flow run on SIGTERM. Either:

• backoffLimit > 0 in which case the K8s retries the job
• backoffLimit = 0 in which case Prefect forces reschedule behaviour on SIGTERM via setting PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR=reschedule

In our case, we have workflows that we want to reschedule with human in the loop when killed due to eviction. That is because processes triggered downstream currently cannot be cancelled and multiple runs of those processes cause all sorts of problems.

The way I've tried to address the problem is by not forcing PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR=reschedule when that var is already set in env.

My perspective of Prefect codebase is very limited at this point so I'd appreciate any guidance on this one.

Checklist

• This pull request references any related issue by including "closes <link to issue>"
  • If no issue exists and your change is not a small fix, please create an issue first.
• If this pull request adds new functionality, it includes unit tests that cover the changes
• If this pull request removes docs files, it includes redirect settings in mint.json.
• If this pull request adds functions or classes, it includes helpful docstrings.

[desertaxle]

Hey @svatopluk-sperka! This looks great! Thanks for taking the time to submit a fix!

[svatopluk-sperka]

That was smooth. Thanks @desertaxle!