Slow page loads under HTTP/1.1 when concurrent uploads are active — migrating from ingress-nginx to NGF v2.4.1/2.5.0

[SJames-Magnet]

HTTP/2 not working through AWS NLB with TLS re-encryption to NGF — migrating from ingress-nginx

Environment

• Cluster: EKS v1.34 (AWS, us-east-1)
• NGF version: v2.5.1 (upgraded from v2.4.1 during debugging)
• Previous ingress: ingress-nginx 1.12.4 (working without issues)
• Load balancer: AWS NLB with TLS termination + TLS target group (re-encryption to backend)
• App: SaaS product using TUS protocol for large file uploads
• NGF replicas: 10 nginx pods, 10 gateway controller pods

Problem

After migrating from ingress-nginx to NGINX Gateway Fabric, page loads become extremely slow (30-70+ seconds) when a file upload (TUS protocol) is running in a separate browser tab. This did not happen with ingress-nginx under the same NLB configuration.

The root cause is that HTTP/2 is not being served to the browser. Under HTTP/1.1, the browser's 6-connection-per-origin limit creates head-of-line blocking when uploads consume connections.

What HAR Analysis Shows

We captured a HAR from the page-load tab while an upload was running in another tab:

• Server response times (wait) are healthy: 300-900ms per request
• All the slowness is browser-side blocked time: requests queuing for 30-70 seconds before being sent
• All requests to our origin are HTTP/1.1 — HTTP/2 is not being negotiated

Request	Total Time	Blocked Time	Server Wait
/api/auth/v0.0/authorization	33,439ms	33,117ms	322ms
/api/auth/v0.0/authorization	34,686ms	34,341ms	345ms
/api/cases/v1.0/cases	71,368ms	70,700ms	664ms
/api/cases/v1.0/cases	72,116ms	71,368ms	748ms

Architecture

Browser ─── HTTP/2 (ALPN h2) ───> AWS NLB (TLS termination, ACM cert)
                                       │
                                       │ TLS re-encryption (target group protocol: TLS)
                                       ▼
                               NGF nginx pods (HTTPS listener, internal cert)
                                       │
                                       │ proxy_pass (mTLS to backend services)
                                       ▼
                               Backend services (:8443)

Ingress-nginx Configuration (Working)

The ingress-nginx NLB has these annotations and works with HTTP/2:

service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:...
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"

• Target group protocol: TLS
• Health check protocol: TCP
• NLB terminates client TLS, negotiates HTTP/2 with browser
• NLB re-establishes TLS to ingress-nginx backend (ALPN h2 negotiated)
• ingress-nginx default_server: ssl_reject_handshake off
• ingress-nginx configmap: use-http2: "true"

What We've Tried on NGF

Step 1: Matched NLB annotations to ingress-nginx

Added to the NGF NginxProxy service patches:

service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP

Result: Target group changed to TLS, all targets healthy, HTTP/2 ALPN negotiated with client. But HTTP/2 stream fails.

Step 2: Changed Gateway listener to HTTPS

listeners:
  - name: "https"
    protocol: "HTTPS"
    port: 443
    tls:
      mode: Terminate
      certificateRefs:
        - name: frontend-gateway-tls
          kind: Secret
    allowedRoutes:
      namespaces:
        from: "All"

No hostname field on the listener (catch-all).

Step 3: Upgraded NGF to v2.5.1

To fix ssl_reject_handshake on on the default_server block (PR #4878). After upgrade, the default_server uses the gateway cert instead of rejecting the handshake.

Step 4: Set disableSNIHostValidation: true on NginxProxy

The NLB connects without proper SNI, causing a Host/SNI mismatch. This setting disables the if ($ssl_server_name != $host) { return 421; } check.

Step 5: Verified HTTP/2 works in-cluster (bypassing NLB)

Connecting directly to an NGF pod by IP address (no SNI, same as NLB behavior):

$ kubectl run test --rm -it --image=curlimages/curl -- \
  curl -vk --http2 --max-time 5 https://10.0.3.88:443/ \
  -H "Host: <<actual hostname>>"

* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
* using HTTP/2
> GET / HTTP/2
< HTTP/2 200

This proves NGF handles HTTP/2 correctly, including when accessed without SNI (v2.5.1 default_server routes by Host header).

Step 6: Verified HTTP/1.1 through NLB works

$ curl --http1.1 https://<<actual hostname>>/
< HTTP/1.1 200 OK

Step 7: Tried disableHTTP2: true on NginxProxy

This removed http2 on; from the NGINX config. Result: broke both HTTP/1.1 AND HTTP/2 through the NLB. Both returned empty responses after 5-second timeout. This proves the NLB backend TLS connection requires NGF to have http2 on for the TLS/ALPN handshake to function correctly.

Step 8: Tested ALPN policy behavior

Set alpn-policy: None — the NLB still served h2 to clients. This confirms that with TLS target groups, the NLB relays the backend's ALPN to the client regardless of the listener ALPN policy setting.

Additional NGINX tuning applied (did not resolve HTTP/2 issue)

• proxy_buffering off via SnippetsFilter
• UpstreamSettingsPolicy — keepalive 320, requests 10000, time 75s
• tcp_nodelay on, ssl_buffer_size 4k
• http2_max_concurrent_streams 256, http2_chunk_size 4k, http2_recv_buffer_size 512k
• ClientSettingsPolicy — client_max_body_size 500m

Current State

With all fixes applied (NGF v2.5.1, TLS target group, TCP health checks, disableSNIHostValidation, hostname-less HTTPS listener):

Test	Result
HTTP/1.1 through NLB	200 OK
HTTP/2 in-cluster to NGF pod (by IP, no SNI)	200 OK
HTTP/2 through NLB	FAILS — "HTTP/2 stream 1 was not closed cleanly"

The NLB successfully:

• Negotiates HTTP/2 with the client (ALPN h2) ✅
• Opens a TLS connection to NGF (target group protocol: TLS) ✅
• Negotiates HTTP/2 with NGF via ALPN on the backend connection ✅

But after sending the HTTP/2 request to NGF, no response comes back — the connection times out after 5 seconds and the NLB sends close_notify:

> GET / HTTP/2
> Host: <<actual hostname>>
>
* TLSv1.2 (IN), TLS alert, close notify (256):
* HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
curl: (92) HTTP/2 stream 1 was not closed cleanly

Config Differences: ingress-nginx vs NGF

Setting	ingress-nginx	NGF	Status
ssl_reject_handshake (default server)	off	on → fixed in v2.5.1	Resolved
proxy_buffering	off	on → off via snippet	Resolved
tcp_nodelay	on	not set → on via snippet	Resolved
ssl_buffer_size	4k	16k → 4k via snippet	Resolved
http2 on	yes (via OpenResty)	yes (global)	Same
Dynamic cert selection	ssl_certificate_by_lua_file	static per-server-block	Different
Server block architecture	single catch-all + Lua routing	per-hostname server blocks	Different
Default server behavior	accepts all TLS, routes via Lua	accepts TLS (v2.5.1), empty block	Different
listen reuseport backlog=4096	yes	no (backlog=511)	Not configurable
aio threads	on + aio_write on	not set	Not configurable

Key Architectural Difference

ingress-nginx uses one catch-all server block with Lua (ssl_certificate_by_lua_file) to dynamically select certificates and route requests. All traffic flows through this single block regardless of SNI.

NGF generates separate server blocks per hostname. With v2.5.1, the hostname-less HTTPS listener provides a cert for the default_server block (so TLS handshakes succeed without SNI), and NGINX routes requests to the correct server block based on the :authority/Host header. This works correctly when accessed directly but fails when the NLB proxies the HTTP/2 connection.

Questions

1. We replicated the exact ingress-nginx NLB configuration (alpn-policy: HTTP2Preferred, backend-protocol: ssl, TLS target group, TCP health checks). HTTP/1.1 works but HTTP/2 fails with stream errors. What about NGF's architecture causes this to fail when ingress-nginx handles it successfully? Is this a known limitation?

2. Is there a recommended architecture for NGF behind AWS NLB that preserves HTTP/2 to the client without TCP passthrough? We cannot manage the public TLS certificate in-cluster.

3. Could the empty default_server block (no locations) be causing the HTTP/2 stream failure? The default_server has the cert but no server_name and no location blocks. When the NLB opens an HTTP/2 connection, could the initial SETTINGS/WINDOW_UPDATE frames be processed by the empty default_server (which has nothing to respond with), causing the timeout?

4. Is there a way to make NGF generate a single server block (similar to ingress-nginx's catch-all pattern) when all HTTPRoutes share the same hostname? Or a way to add routes/locations to the default_server block?

5. Could the NLB's HTTP/2 behavior differ based on target group creation method? Our target group was transitioned from TCP to TLS (via annotation change on an existing service). Ingress-nginx's was created as TLS from the start. Could this affect HTTP/2 proxying behavior?

6. Are there any NginxProxy or Gateway settings we've missed that could affect HTTP/2 connection handling between the NLB and NGF?

Reproduction Steps

1. Deploy NGF v2.5.1 behind AWS NLB with:
   • aws-load-balancer-ssl-cert + aws-load-balancer-ssl-ports: "443" (TLS termination)
   • aws-load-balancer-backend-protocol: ssl (TLS target group)
   • aws-load-balancer-alpn-policy: HTTP2Preferred
   • aws-load-balancer-healthcheck-protocol: TCP
2. Configure Gateway with hostname-less HTTPS listener + cert
3. Set disableSNIHostValidation: true on NginxProxy
4. Attempt curl --http2 https://<hostname>/
5. Observe: HTTP/2 ALPN negotiated, request sent, 5-second timeout, close_notify

Any guidance would be greatly appreciated.