Requeue if create pod returns already exists error

[nikola-jokic]

There are situations when the newly created pod is not yet delivered through the watcher, causing subsequent reconciliations to issue a second create pod event.

This change tests for the condition, logs at the info level, and re-queues the event in 5s.
It is not necessary to do it for secrets since the secrets are not watched; therefore, the GET request was being issued directly.

It will reduce the noise in the log while backing off the reconciliation for a bit.

[Copilot]

Pull Request Overview

This PR addresses a race condition where the ephemeral runner controller attempts to create a pod that already exists but hasn't been received through the Kubernetes event watcher yet. The change adds error handling to detect "already exists" errors and requeues the reconciliation with a 5-second delay instead of logging an error.

• Adds detection for "already exists" errors when creating pods
• Implements a requeue mechanism with 5-second delay for the race condition
• Updates log message to be more descriptive about the pod creation process

[Copilot]

The log message is misleading. At this point, the code has determined the pod doesn't exist, but the new error handling suggests the pod might actually exist. Consider a more accurate message like 'Attempting to create new ephemeral runner pod'.

Suggested change

	log.Info("Ephemeral runner pod does not exist. Creating new ephemeral runner")
	log.Info("Attempting to create new ephemeral runner pod")

[erinhannum]

@nikola-jokic Do we have an updated ETA on when we could expect this to be deployed to prod? Is there anything I can help unblock to expedite? Thanks!
cc: @kevinegordon03

[otfung]

@nikola-jokic , which version will include this PR#4201? Any ETA? Thanks!
cc: @kevinegordon03, @erinhannum

[nikola-jokic]

Hey @otfung,

This will be included in 0.13.0. The deadline for the next release is 13th of October, but it might be sooner, depending on the progress of the rest of the work planned for the 0.13.0 release