Introduce a context for canceling wait operations

[matheuscscp]

Closes: #31382

The behavior expected for a release with a custom wait context that gets done during a wait operation is expected to be the same as the behavior for a timeout (which is the default behavior).

[matheuscscp]

@joejulian I think you mentioned in the dev meeting that some changes are required? Can you please clarify? 🙏 Thanks!

[matheuscscp]

@joejulian @gjenkins8 @mattfarina I added tests, please review and let me know if anything else is missing 🙏

[matheuscscp]

@gjenkins8 Fixed both comments, thanks! PTAL one more time 🙏

[matheuscscp]

@joejulian I was reflecting about what you said in the last dev meeting and I think I understood what you meant. Upon a wait timeout, the release is left in the pending state i.e. pending-install, pending-upgrade or pending-rollback.

Flux has dealt with this a long time ago: we detect this pending state and "unlock" the release to keep moving forward. A wait timeout is a very common issue and Flux couldn't leave this unattended for so long. Here's how we do it:

Detection:

1. https://github.com/fluxcd/helm-controller/blob/96f1d9e1e28c3228d27524a5c28457df412c12e3/internal/reconcile/state.go#L89-L105
2. https://github.com/fluxcd/helm-controller/blob/96f1d9e1e28c3228d27524a5c28457df412c12e3/internal/reconcile/atomic_release.go#L375-L377

Resolution: https://github.com/fluxcd/helm-controller/blob/96f1d9e1e28c3228d27524a5c28457df412c12e3/internal/reconcile/unlock.go#L55-L91

So, as I said before, from the perspective of us SDK users, the feature being introduced in this PR is merely "a faster/smarter/customized way for producing timeouts on wait operations". It's beyond the scope of this PR dealing with timeouts.

I think with this we can lift the embargo here, what do you think? @joejulian @gjenkins8 @mattfarina @scottrigby @stealthybox @stefanprodan

Furthermore, I have backported the legacy waiter changes here to a fork from v3.19.0 and built an e2e PoC for the Flux feature, which is working as expected. I created a Flux HelmRelease with a bogus image tag, causing Helm to get stuck in a wait operation with timeout set to 3m due to the pods in ErrImagePull. When the Flux HelmRelease .spec.values.image.tag is updated to a valid tag, I immediately see this Kubernetes event emitted by helm-controller:

- apiVersion: v1
  count: 1
  eventTime: null
  firstTimestamp: "2025-11-01T12:13:44Z"
  involvedObject:
    apiVersion: helm.toolkit.fluxcd.io/v2
    kind: HelmRelease
    name: podinfo
    namespace: frontend
    resourceVersion: "92444307"
    uid: 29c2a513-5662-42b6-afe4-ed157a5c60f2
  kind: Event
  lastTimestamp: "2025-11-01T12:13:44Z"
  message: |-
    Helm upgrade failed for release frontend/podinfo with chart [email protected]+971fef0d04d5: New reconciliation triggered by HelmRelease/frontend/podinfo

    Last Helm logs:

    2025-11-01T12:12:50.46219424Z: Looks like there are no changes for Service "podinfo"
    2025-11-01T12:12:50.477013785Z: Patch Deployment "podinfo" in namespace frontend
    2025-11-01T12:12:50.494020691Z: Looks like there are no changes for Ingress "podinfo"
    2025-11-01T12:12:50.508865623Z: Patch ServiceMonitor "podinfo" in namespace frontend
    2025-11-01T12:12:50.526101352Z: waiting for release podinfo resources (created: 0 updated: 4  deleted: 0)
    2025-11-01T12:12:50.526105791Z: beginning wait for 4 resources with timeout of 3m0s
    2025-11-01T12:12:50.550119838Z: Deployment is not ready: frontend/podinfo. observedGeneration (22) does not match spec generation (23).
    2025-11-01T12:12:52.547186135Z: Deployment is not ready: frontend/podinfo. 0 out of 1 expected pods are ready
    2025-11-01T12:13:42.540396144Z: Deployment is not ready: frontend/podinfo. 0 out of 1 expected pods are ready (24 duplicate lines omitted)
    2025-11-01T12:13:44.349242962Z: wait for resources failed after 54s: context canceled
    2025-11-01T12:13:44.367689523Z: warning: Upgrade "podinfo" failed: context canceled
  metadata:
    annotations:
      helm.toolkit.fluxcd.io/app-version: 6.9.2
      helm.toolkit.fluxcd.io/oci-digest: sha256:971fef0d04d5b3d03d035701dad59411ea0f60e28d16190f02469ddfe5587588
      helm.toolkit.fluxcd.io/revision: 6.9.2+971fef0d04d5
      helm.toolkit.fluxcd.io/token: sha256:1bb60c0d9974502e0cbea5486437617fcec597d9c99f034796135fdabf91b062
    creationTimestamp: "2025-11-01T12:13:44Z"
    name: podinfo.1873e0f58808d7bd
    namespace: frontend
    resourceVersion: "92444900"
    uid: 2a338055-6054-4038-8072-44c389311d4b
  reason: HealthCheckCanceled
  reportingComponent: helm-controller
  reportingInstance: ""
  source:
    component: helm-controller
  type: Warning

And immediately after this event appears, helm-controller reconciles the Flux HelmRelease once again, this time successfully:

NAME                      READY   STATUS    RESTARTS   AGE
podinfo-5564f7698-bmtld   1/1     Running   0          4m4s
podinfo-5564f7698-wgpc8   1/1     Running   0          2d5h

NAME      AGE   READY   STATUS
podinfo   27d   True    Helm upgrade succeeded for release frontend/podinfo.v28 with chart [email protected]+971fef0d04
d5

This shows that Flux is correctly dealing with the release left on the pending state, i.e. the same code path that deals with wait timeouts is kicking in.

[joejulian]

Sure, that's fine with me. I just wanted to make sure we had an expected outcome that can be documented.

[gjenkins8]

Changes here -- introducing a context for the wait operation (that can be canceled by the caller) -- look good to me.

Additional behavior around what to do with the release once the wait has been canceled, remains/is the callers responsibility (and hopefully Helm can improve here over time too)

Authors have tested the changes above (via flux)

[scottrigby]

lgtm!

[joejulian]

It's already been approved and merged, but I'll still add my posthumous (not sure if that's a good word for this) approval.