Support TLS for all communication paths

[elevran]

What would you like to be added:
Support encryption on inference gateway communication paths.

1. Gateway to EPP (EPP TLS support provides very minimal protection #582)
2. Gateway to model server

At a minimum include documentation on how to configure the system for end-to-end encryption. Optionally, include this as part of the IGW configuration or contract with gateways.

Why is this needed:
Enterprise requirement and regulations may dictate use of TLS to secure communications inside cluster.
This can be enabled by deploying a service mesh, which might be a relevant option in some deployments.
In other scenarios, IGW might need to support encryption directly.

• Communication to EPP can already be secured (albeit with limitations - see EPP TLS support provides very minimal protection #582).
• EPP to model server (e.g., solved for metricsscraping by specifying the scheme viamodel-server-metrics-scheme` CLI option)
• since metrics and inference are served over the same port, the use of the above also implies inference should be over TLS.

In the case of HTTP client requests, the gateway would need to upgrade the connection to model servers to use HTTPS.
There is no indication in InferencePool or EPP response (EPP returns IP:port to the gateway, not scheme) whether model servers serve over HTTPS.
Unfortunately, BackendTLSPolicy mandates a Service backend and can't be used for InfererncePool backends. Gateway specific configurations could be used in some cases (e.g., Istio DestinationRule) - but are implementation dependent and place responsibility with the user (e.g., a "shadow service" is defined for the inference pool, but its name is computed at runtime).

[smarterclayton]

@robscott

[elevran]

Per Slack comment, the docs for BackendTLSPolicy here (...only Service is allowed) are not up to date. Updated docs are here. Implementation may implement the policy for non-Service objects.
Thus, this issue might only require testing, example documentation (e.g., in a blog) and opening issues with specific Gateway implementations.

[robscott]

BackendTLSPolicy is about to graduate to GA in Gateway API v1.4, thanks in large part to @candita, @kl52752, and @snorwin. As @elevran mentioned, it seems like BackendTLSPolicy is the correct solution here. While Service is the only target supported broadly by implementations today, we should provide a recommendation for how InferencePool should be supported. This will not require any API change, just implementation support.

In general, I'd expect to have 2 different targets here:

1. The Endpoint Picker Service for EPP endpoints
2. The InferencePool for model server endpoints

It's possible that a single BackendTLSPolicy could be used for both, but I'd expect the common case to have separate policies for each. I know @kl52752 is working on BackendTLSPolicy support in GKE, @danehans would likely have the best answer for Kgateway. There are lots of folks working on Istio, maybe @keithmattix or @howardjohn would have some ideas about support here.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elevran]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[elevran]

@Gregory-Pereira is TLS a priority?

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten