Data plane serves stale upstream endpoints after backend rollout → intermittent 502 (no live upstreams) until NGINX pod restart

[SidharathBansal]

After a routine, healthy backend Deployment rollout, a subset of our NGINX data-plane pods kept routing to the old (terminated) upstream pod IPs. The control plane failed to push the updated endpoints to those specific data-plane pods, so NGINX marked every upstream for the affected host as down and returned intermittent 502 (no live upstreams, plus 113 Host is unreachable / 110 Operation timed out against the dead IPs).

Because only some data-plane pods were stale and the load balancer spreads traffic across all of them, the failures were intermittent (retries that landed on an up-to-date pod succeeded). The condition did not self-heal — it only cleared when the data-plane pods were restarted (in our case an automatic rollout restart), after which fresh pods pulled current config and served the exact same backends with zero upstream errors.

On the control plane, the config delivery to the affected agent(s) was failing around the incident with:

"msg":"Failed to update NGINX configuration","error":"context canceled"
"msg":"error sending request to agent","error":"context canceled"
"msg":"error setting initial configuration","error":"rpc error: code = Canceled desc = context canceled"

This looks like the same failure class as #5330 (stale IP / upstream errors after a rollout that spreads and is cured by a restart), but we are on 2.4.2, not 2.6.1, and our control-plane marker is context canceled on config apply rather than connection not found. Filing separately with 2.4.2 evidence so it can be confirmed whether the same fix applies or a 2.4.x-specific issue exists.

To Reproduce

Observed in production during a normal rollout; conditions:

1. Deploy NGF (OSS data plane) with the control plane scaled to multiple replicas (we run 2 control-plane + 5 data-plane pods) fronting a Service via an HTTPRoute (single host, PathPrefix: /).
2. Let the gateway run for a while under real traffic (our affected data-plane pods had been up ~1 day).
3. Roll the backend Deployment for that host (new pods come up on new IPs; old pods terminate) — the rollout itself is healthy.
4. Observe that some data-plane pods keep dialing the old upstream IPs and return no live upstreams / 502 for that host, while other data-plane pods serve fine → intermittent 502 at the edge.
5. The affected pods do not recover on their own. Restarting the NGINX data-plane pods clears it immediately.

We also have a churn-based reproduction harness that drives heavy EndpointSlice churn (KEDA cron swinging a 700-pod backend 700⇄1000 under ~50–70k RPM of k6 load for 1h) to surface reload/stale-endpoint 502s on the OSS data plane.

Expected behavior

After a backend rollout, all data-plane pods should converge to the current endpoints. If the control plane fails to apply config to an agent (e.g. context canceled on the gRPC stream), it should retry until the data plane is in sync and/or the data-plane pod should not be left silently serving stale upstreams indefinitely. A data-plane pod with an all-dead upstream set should not require a manual/rollout restart to recover.

Your environment

• Version of the NGINX Gateway Fabric: 2.4.2 (control-plane image .../nginx-gateway-fabric:2.4.2; data-plane image .../nginx-gateway-fabric/nginx:2.4.2)
• NGINX data plane: OSS
• Version of Kubernetes: EKS (v1.34.x)
• Kubernetes platform: AWS EKS
• How NGF is exposed: Gateway fronted by an AWS ALB → NGINX Service → HTTPRoute → backend Service
• Topology: 2 control-plane replicas + 5 data-plane pods (data-plane count ≥ control-plane count, per the scaling docs)
• Agent: nginx-agent v3

Logs

Data plane (host and IPs sanitized) — the affected pods returning stale-endpoint errors after the backend rollout:

[error] connect() failed (113: Host is unreachable) while connecting to upstream, server: admin.example.com, upstream: "http://10.0.0.11:80/...", host: "admin.example.com"
[error] upstream timed out (110: Operation timed out) while connecting to upstream, server: admin.example.com, upstream: "http://10.0.0.12:80/...", host: "admin.example.com"
[error] no live upstreams while connecting to upstream, server: admin.example.com, upstream: "http://admin-backend/...", host: "admin.example.com"

Key observation from a ~4-minute error export spanning the incident + the restart boundary:

• Every error line for the affected host came from the OLD data-plane pods (pod-template-hash A). Zero errors came from the NEW pods (pod-template-hash B) that came up after the restart, even though both served the same backend.
• For the affected host, NGINX was dialing a fixed set of 5 upstream IPs and hitting no live upstreams — i.e. all upstreams in NGINX's table were the pre-rollout, now-terminated pod IPs.
• Error breakdown for the affected host: no live upstreams ×296, 113 Host is unreachable ×90, 110 Operation timed out ×80.

Control plane (around the incident):

{"level":"error","logger":"eventHandler","msg":"Failed to update NGINX configuration","error":"context canceled", ... "handler.go:298"}
{"level":"error","logger":"nginxUpdater.commandService","msg":"error sending request to agent","error":"context canceled", ... "command.go:415"}
{"level":"error","logger":"nginxUpdater.commandService","msg":"error setting initial configuration","error":"rpc error: code = Canceled desc = context canceled", ... "command.go:319"}

Additional context

• Timeline: backend rollout → intermittent 502 within seconds → persisted ~11 min → auto rollout-restart of the NGINX data-plane Deployment → 502s stopped once the fresh pods synced config (~2 min later). No 502s from the post-restart pods.
• Impact was intermittent precisely because only a subset of the 5 data-plane pods were stale; the ALB distributed traffic across all 5.
• Multiple control-plane replicas note: with 2 CP replicas, agents connect to both, and only the leader writes status. If an affected data-plane pod is connected to a non-leader and config apply fails, that error is not reflected in Gateway/HTTPRoute status — so resource status looked healthy while the data plane served stale upstreams. Please advise whether multi-replica CP + agent stream context canceled can leave a data-plane pod wedged without a retry that re-syncs it.
• Relation to New upstream 110 error against stale IP after 2.6.1 #5330: same symptom/mechanism (stale upstream IPs after rollout, no live upstreams/110/113, cured by restart). We're on 2.4.2 (not the 2.6.1 regression noted there), so it would help to confirm whether the v2.7.0 fix covers this path or whether 2.4.x needs a separate fix.
• Questions for maintainers:
  1. When the agent config stream returns context canceled during setInitialConfig / config apply, does the control plane guarantee a subsequent successful re-apply, or can a data-plane pod be left permanently stale until restart?
  2. Is there a recommended readiness/liveness or self-heal so a data-plane pod with an all-dead upstream set recovers without a manual restart?
  3. Does running multiple control-plane replicas increase the likelihood of this wedged-agent state, and is single-replica CP safer for 2.4.x until this is resolved?
  4. is this bug already resolved in 2.6.6?

[nginx-bot]

Hi @SidharathBansal!

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.