Remove ephemeral runner when exit code != 0 and is patched with the job

[nikola-jokic]

If the runner exits with the exit code != 0, it might have been killed by something external.

Therefore, we should delete this ephemeral runner without having to retry it since its configuration is no longer valid, and the job failed.

Regarding #4091, 0.12.1 release should fix the issue where the race condition is not checked when the runner exits with exit code 0. However, if there is pressure on the node that kills the runner, we would still retry the runner even though the credentials are no longer valid. This PR aims to fix it completely by removing the ephemeral runner IF we are certain that the job was assigned to it.

Let's explain the worst case scenario:

1. The ephemeral runner is created and started the pod.
2. The pod starts and is killed right away
3. At this point, the listener did not have time to patch the ephemeral runner just yet.
4. The ephemeral runner controller would re-create the job pod with the back-off.
5. The listener during this time should already be notified about the job assignment, and should patch the ephemeral runner.
6. The pod exits again, since the configuration is no longer valid
7. However, this time, ephemeral runner controller notices that the job was assigned to that ephemeral runner, and removes the ephemeral runner, cleaning up the slot for scaling.

Fixes #4091

[Copilot]

Pull Request Overview

This PR changes the behavior of ephemeral runners to immediately delete them when they exit with a non-zero exit code and have a job assigned, rather than retrying. This addresses scenarios where the runner may have been killed externally, making its configuration invalid.

• Replaces job assignment check with a HasJob() method for better code clarity
• Modifies failed runner behavior to delete ephemeral runners with jobs instead of retrying
• Updates successful completion behavior to delete runners immediately instead of marking as finished

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
ephemeralrunnerset_controller.go	Replace direct JobRequestId check with HasJob() method call
ephemeralrunner_controller.go	Add logic to delete ephemeral runners on failure when job assigned, remove markAsFinished method
ephemeralrunner_controller_test.go	Add test for deletion behavior on failure with job, update existing tests
ephemeralrunner_types.go	Add HasJob() helper method to check if runner has assigned job

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

[nitpick] Variable er is being reassigned with a new instance unnecessarily. The existing instance can be reused since you're fetching fresh data from the API server in the Eventually block.

[Copilot]

[nitpick] Variable er is being reassigned with a new instance unnecessarily. The existing instance can be reused since you're fetching fresh data from the API server in the Eventually block.