Spike: Test new ingress/gateway controller

[Richard87]

Make sure it supports our needs:

• Slow server responses (over 30sec, Read/Send timeouts): timeouts are disabled by default. If timeouts are set they are for the full lifecycle of the request, which is different from ingress nginx where timeouts are between successive reads/writes. Response details, e.g. timeouts, are logged in gateway pod in field response_code_details and response_flags,
• Long running websockets ( up to 60min, with/without ping-pongs ): web sockets does not seem to be affected by request timeouts. Setting a low timeout causes normal http requests to fail, but web socket connections works fine even when ping/pongs are set to a value higher than request timeout. We can therefore safely disregard the websocketfriendly ingressConfiguration
• Large body sizes: tested upload of 3GB file without and issues
• Proxy Buffering: not supported
• Sticky Sessions: currently not implemented in Istio, ref Implement SessionPersistence in Gateway API istio/istio#55839. See comment
• Server Sent Events
• Metrics
• Log (json)
• Enforce http->https redirection works for ALL routes (default http route for port 80), Prefere one global rule (example)
• Ensure same hostname cannot be used across applications (admission webhook?)
• Ingress: IP Allow list: not supported
• Test Client Certificate Authentication - https://kubernetes.slack.com/archives/CR0H13KGA/p1765547533826529 Currently we have 3 options:
  1. Give the user their own gateway, loadbalancer and IP with ClientCert config
  2. Configure TLS Passtrough, mount certificate secret into pod, let Application manage tls termination and validation themselves (possibly running a nginx proxy)
  3. Not supporting the feature / Waiting for ListenerSet to support it natively: I think we should opt for this: three apps have configured it, and we should inform then that this feature will be removed
• Set HSTS on all responses (Strict-Transport-Security header)
• Cert Mananger configure towards gateway (docs), request Digicert Staging account
• Operator
  • Configure Network Policy (configure allowed namespaces in helm values and put in radix-deny-traffic-from-other-ns created by operator)
  • Enforce oauth2 service in proxy mode
  • Create single httproute pr component for all hostnames with reference to common gateway and component-specifcic-listener.
• Load balancer k8s service: currently uses hardcoded ip, should use named azure ip resource as annotation service.beta.kubernetes.io/azure-pip-name (docs)
• Migration
• Migrate: to migrate part of the traffic
  • Migrate specific apps by hand upfront to teste responses and rewquest, status codes etc. Consider using DNS Loadbalancing
• Load testing with long running requests, changing routes, slow responsnes, high volume

Ignoring tasks:

• Remove response Server header + other istio specific headers (?) No, its public anyway in radix-flux/operator
• Should we use ExternalDNS to allow users to test new ingress controller before being forced over? No, we assume they dont care to test, and it should just work anyway
• Not feasable for now: Default backend (503) (Would be amazing if we can return the body from the backend if it is included, but only fallback to our default backend if its blank)

[Richard87]

Istio

XListenerSet issues, possible cause? istio/istio#58019
1.27.1 works
1.27.2 works/fails (Rolling up from 1.27.1 keeps working, reinstalling resources in 1.27.2 fails, coming from 1.27.3/1.28.0 fails)
1.27.3 fails
1.28.0 fails

Samples

https://github.com/istio/istio/blob/master/samples/httpbin/gateway-api/httpbin-gateway.yaml

1.28.0

Gateway -> XListenerSet -> HttpRoutes

Gateway: HTTP 0 attached route, but works
Gateway: HTTPS 1 attached routes, fails
XListenerSet: HTTP 0 attached routes, but works
XListenerSet: HTTPS 0 attached routes, fails

[nilsgstrabo]

radix-operator:

• configure networkpolicy radix-deny-traffic-from-other-ns to allow traffic from istio-system. We should make this configurable via helm values.yaml, perhaps by exposing the full networkpolicy spec or just the spec.ingress.
• deployment reconciler:
  • shall we create gateway resources (xlistenerset, httproute) for all apps, or only on opt-in.
  • How will user opt-in to route traffic through istio? annotations on radixconfig? And how can we automate DNS zone entries to send traffic to istio LB?
  • Check timeout for web sockets.
  • Check how we can map existing network.ingress.public in radixconfig, and ingressConfigurations defined in configmap radix-operator-ingress-configmap, to istio

Misc:

• HSTS: All responses must have Strict-Transport-Security header
• Remove response Server header + other istio specific headers (?)
• Ensure that http->https redirection works for ALL routes

[nilsgstrabo]

sessionAffinity is not supported (yet) by Istio, and twelve apps in platform1 have defined it in radixconfig. We need to check with the teams if this is something they still need.

[Richard87]

Istio works great