Interaction between TLS config on HTTPRoute and Gateway

[mark-church]

After trying to implement TLS in the GKE Gateway controller it raised some questions around what the interaction should be between TLS config in the HTTPRoute and the Gateway. Let's say I have the following Gateway:

kind: Gateway
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: tls-gateway
spec:
  gatewayClassName: acme-lb
  listeners:
  - protocol: HTTPS
    port: 443
    tls:
      certificateRef:
        kind: Secret
        group: core
        name: gateway-cert
      routeOverride:
        certificate: Allow
    routes:
      kind: HTTPRoute
---
kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: foo1
spec:
  hostnames:
  - "foo.example.com"
  tls:
    certificateRef:
      kind: Secret
      group: core
      name: foo-cert-1
  rules:
  - matches:
    - path:
        type: Prefix
        value: /path1
    forwardTo:
    - serviceName: foo-svc-1
      port: 8080
---
kind: HTTPRoute
apiVersion: networking.x-k8s.io/v1alpha1
metadata:
  name: foo2
spec:
  hostnames:
  - "foo.example.com"
  tls:
    certificateRef:
      kind: Secret
      group: core
      name: foo-cert-2
  rules:
  - matches:
    - path:
        type: Prefix
        value: /path2
    forwardTo:
    - serviceName: foo-svc-2
      port: 8080

This config raises a couple gray areas:

• How does a Gateway certificate without a hostname specified, interact with HTTPRoutes that have certificates and hostnames? Does it become like a default certificate that terminates any HTTPRoutes on the listener that don't have certificates? Or is TLS only supported on the Gateway OR HTTPRoute but not both?
• What if the Gateway does have TLS and a hostname specified? Would that TLS config take precedence over a Route with the same hostname and a TLS config? Or would the HTTPRoute take precedence?
• How should conflicts be handled between two HTTPRoutes with different certificates but overlapping hostname? Is this simply a bad practice and should result in error?
• If an HTTPRoute does error due to cert conflict, does the HTTPRoute fail-closed and the Route doesn't get applied at all? This is probably implementation specific but just wondering.

cc @hbagdi @robscott @bowei

[SantoDE]

Just my quick 2 cents:

How does a Gateway certificate without a hostname specified, interact with HTTPRoutes that have certificates and hostnames? Does it become like a default certificate that terminates any HTTPRoutes on the listener that don't have certificates? Or is TLS only supported on the Gateway OR HTTPRoute but not both?

I would guess, it should be some sort of default certificate for HTTPRoutes that don't have own certificates. Then, the cluster operator still would have control over that in order to protect all routes bound to that listener with TLS?

What if the Gateway does have TLS and a hostname specified? Would that TLS config take precedence over a Route with the same hostname and a TLS config? Or would the HTTPRoute take precedence?

For me, thats a question of the roles. Currently, I would guess Gateway definitions take precedence as its probably set by the cluster operator?

How should conflicts be handled between two HTTPRoutes with different certificates but overlapping hostname? Is this simply a bad practice and should result in error?

We had similiar questions on Traefik and didn't found a proper answer yet. For now, its "just" picking one as I tend to agree its bad practice.

[mark-church]

@stevesloka @danehans @hbagdi - how are Contour and Kong planning to deal with this (interaction between Gateway and HTTPRoute secret)?

Here is one proposal, but I can think of other combinations that would also make sense so feel free to shoot holes in this:

• The secrets specified for a Gateway listener are used for any HTTPRoutes bound to it, if the HTTPRoutes don't have secrets specified themselves
• The secrets specified for a Gateway listener are used for any HTTPRoutes bound to it if routeOverride.certificate=Deny (this is also the default)
• If routeOverride.certificate=Allow and a bound HTTPRoute has secrets configured, then the HTTPRoute's secrets will be used to terminate traffic for hostnames in the HTTPRoute
  • Conflicting Secrets between HTTPRoutes that match on the same hostname is an error, but it may be difficult for the controller to flag this error since it would have to look at the secret.

[stevesloka]

I think Contour is in the same boat as @SantoDE is where we have't gotten there just yet, but I your proposal @mark-church outlined makes sense to me.

The other scenario I'm not sure on would be:

• A Gateway has a secret specified, routeOverride.certificate=Allow is set, but two different HTTPRoutes reference different secrets for the same host.

I'm not sure how to validate that since you'd have conflicting secrets for the same resource fqdn, or would you have to use the timestamps again to pick the oldest cert? What error Condition could we raise since one of the secrets is going to get ignored?

[SantoDE]

Yeah @stevesloka, we are in the same boat :)

And I also agree that @mark-church's purpose is good. About validation that cert from @stevesloka, I think we would either have to pick the oldest, or error out the Gateway with a config issue condition? 🤷‍♂️

[mark-church]

It's a long shot but I'm just pinging some of the top cert-manager contributors here on this question to see if they have thoughts on this topic. Hi @munnerz, @JoshVanL, and @meyskens - Gateway is an evolution to the Ingress API being worked on in SIG-Network. There are two places where TLS can be configured - on the HTTPRoute and also on the Gateway and we are discussing how these two TLS configurations should interact, override, supercede, default, etc between each other as in the following picture. Since you're much closer to TLS workflows and user journeys I figure you might have an interesting take on this.

[meyskens]

Hi 👋 I have been quickly reading into the whole proposal to get an idea how it functions. As i understand a Gateway would handle multiple routes which each also have their own TLS cert. I do wonder why both have a TLS cert? The example in the issue the two HTTPRoutes have different certificates but the same domain, which is not possible as the gateway will only receive he HTTP Path after the TLS handshake is finished. Which can cause confusion which certificate is used... that should be avoided at all costs (an error?) That becomes harder cross-namespace.

What if the Gateway does have TLS and a hostname specified? Would that TLS config take precedence over a Route with the same hostname and a TLS config? Or would the HTTPRoute take precedence?

If the Gateway has domain specific certs which can be specified i think it should use those, giving the operator control over the exact certificate surfaced to the client and avoiding the per path certificate scenario (thinking of it Ingress has that today also...).

I think I ended up with more questions that with a good answer, sorry :D

[SantoDE]

👋

As i understand a Gateway would handle multiple routes which each also have their own TLS cert. I do wonder why both have a TLS cert?

It's a question of responsibilities out of the gateway spec so far, I would assume. One is controlled by an operator while the other might be merely maintained by a developer. For me, the general idea would be that a gateway could potentially set some defaults or more "important" certs, but who knows?

If the Gateway has domain specific certs which can be specified i think it should use those, giving the operator control over the exact certificate surfaced to the client and avoiding the per path certificate scenario (thinking of it Ingress has that today also...).

I agree with you. If there is a matching cert on a gateway, it should take those. On Routes, it should only be used if none matching can be found at the gateway level (imho).

[meyskens]

It's a question of responsibilities out of the gateway spec so far, I would assume. One is controlled by an operator while the other might be merely maintained by a developer. For me, the general idea would be that a gateway could potentially set some defaults or more "important" certs, but who knows?

I would always assume the operator has the last say :) It does solve a common issue of needing 1 certificate across namespace boundaries.

[bowei]

Some notes:

Gateway provider has to explicitly allow override (see RouteOverride field) to allow the behavior -- and this represents a delegation of authority w/ attendant issues once it has been delegated. It was discussed that for more complex permissions model, this can be implemented via separate infra that is trusted w/ edit access to the Gateway itself. Sketch: set of CRs accessible to the *Route authors that handles allocation of certs and domains based on some criteria.

I know that K8S grouped them together, but certs are public and can have conflicting SANs detected by inspection and hence the controller can perform validation:

• Controller can read secrets -- which means that it can perform full validation
• a different infra is handling certs+secrets -- I assume that most infra will have a way to view the cert domains etc w/out needing access to secrets in this case. Note that doing this properly is up to the impl, it's hard to definitively enforce this at the API schema level.

There is nothing that requires the private key to be bundled in the same visibility scope as the certificate.

[howardjohn]

Putting in my 2c:

Overarching question I have, which is a really hard but important question: Do we support traffic where the SNI and Host header are not the same? This would apply to HTTPS only

How does a Gateway certificate without a hostname specified, interact with HTTPRoutes that have certificates and hostnames? Does it become like a default certificate that terminates any HTTPRoutes on the listener that don't have certificates? Or is TLS only supported on the Gateway OR HTTPRoute but not both?

Seems like route ones are called overrides, so if its in Route it takes precedence. Any routes with TLS config will present those certificates when SNI matches route.hostname. Note that this means that SNI must equal Host!

Open question: for a request to foo.bar, do we do a TLS handshake (then 404, due to no matching routes) or not accept the connection (due to no matching SNI)? This goes back to the original question

What if the Gateway does have TLS and a hostname specified? Would that TLS config take precedence over a Route with the same hostname and a TLS config? Or would the HTTPRoute take precedence?

No, HTTPRoute is always priority

How should conflicts be handled between two HTTPRoutes with different certificates but overlapping hostname? Is this simply a bad practice and should result in error?

This is a conflict - even without TLS. I think the ordering is already defined on how conflicts are handled?

If an HTTPRoute does error due to cert conflict, does the HTTPRoute fail-closed and the Route doesn't get applied at all? This is probably implementation specific but just wondering.

Per conflict resolution, I think one of the routes "wins" and is used fully while the other is completely ignored (with some error in the status)

[hbagdi]

I think most of these cases were discussed when we merged TLS but they are not explicitly documented.

How does a Gateway certificate without a hostname specified, interact with HTTPRoutes that have certificates and hostnames? Does it become like a default certificate that terminates any HTTPRoutes on the listener that don't have certificates? Or is TLS only supported on the Gateway OR HTTPRoute but not both?

Certificate without hostname at Gateway implies a default certificate. TLS is supported on Gateway AND XRoute at the same time (iff routeOverride.Certificate is set to true).

What if the Gateway does have TLS and a hostname specified? Would that TLS config take precedence over a Route with the same hostname and a TLS config? Or would the HTTPRoute take precedence?

HTTPRoute takes precedence. This was to solve use-cases like #103.

How should conflicts be handled between two HTTPRoutes with different certificates but overlapping hostname? Is this simply a bad practice and should result in error?
If an HTTPRoute does error due to cert conflict, does the HTTPRoute fail-closed and the Route doesn't get applied at all? This is probably implementation specific but just wondering.

Follow Conflict resolution guidelines. The overlapping or conflicting certificate is a problem with Ingress v1 and v1beta1 too and no one has figured out a good solution AFAIK.

Open question: for a request to foo.bar, do we do a TLS handshake (then 404, due to no matching routes) or not accept the connection (due to no matching SNI)? This goes back to the original question

This probably will be up to implementations. I'm not sure what the general practice here.

The most important take away is that we need to educate users about the routeOverride.certificate field and its implications and then advise to use OPA for further enforcement. As Bowei said, not much can be done at the schema level here.

[louiscryan]

Reading through linked issues it seems like there are two tensions in the API:

• (A) Who gets define a VHOST and it's properties
• (B) The difference between where a cert is loaded from and what properties that cert or TLS negotiation should have.

The model in proxies is LISTENER(IP, port) --> VHOST(domain/host, TLS) --> ROUTE(host, path, headers, ...)

The discussion in #103 is really about allowing app-owners to define VHOST if I'm reading it right. We think sometimes the ownership model is infra-admin(LISTENER, VHOST) & app-admin(ROUTE) and sometimes it's infra-admin(LISTENER) & app-admin(VHOST, ROUTE). Do folks have any 80/20 rules about the prevalence of one situation or the other?

My experience has been that most enterprises use broad wild-carded VHOSTs owned by infra-admin and that some large solution-providers (think wordpress) use a VHOST-per-tenant. I see more instances of the former but the instances of the latter tend to be quite large.

In the API model Gateway=LISTENER+VHOST and we didn't want app-owners to control LISTENER properties in shared infra. Since Gateway-->XXXRoute supports delegation from infra-admin to app-owner the solution to the above was to start to duplicate aspects of VHOST into XXXRoute resources to leverage that delegation. This choice came with the following costs:

• TLS configuration space can be quite large so this can become a large duplicate surface
• Any VHOST capable protocol added via a new XXXRoute would need to duplicate what was done for HTTP. E.g. SMTP, CoAP, AMQP.
• TLSRouteConfig is delegating the 'source' of the PKI which is different from 'constraints on the behavior of TLS and PKI selection'. This is important when PKI is provided as a service outside of the K8S secret model.

Alternative API models would have been:

1. Gateway Merging. Leave VHOST in Gateway and allow for Gateway merging to enable VHOST delegation to app-owner. May be needed anyway by infra-owners to manage port/IP ownership but would require a constraint on app-owner ability to override LISTENER so requires some adjustment to API.
2. Decompose VHOST into separate resource and allow for VHOST delegation in the same manner as route. Gateway --> VHost --> XXXRoute. Probably worth noting that VHOST and TLSRoute could possibly become the same API (Gateway --> VHost --> Service)
3. Hybrid of above with in-line VHOST in Gateway as is currently defined to short-hand simple, i.e broad wildcard or small host cardinality, use-cases

[robscott]

Thanks for the great write up @louiscryan! I think you've summarized our current state and tradeoffs well. I just have a couple questions about the alternatives you mention at the end:

1. What would it look like to "Leave VHOST in Gateway and allow for Gateway merging to enable VHOST delegation to app-owner"? Would this involve infra-owners owning a root Gateway that could then delegate domains to Gateways owned by app-owner?
2. I'm having trouble picturing how the third option would differ from what we already have, do you have an example of what that might look like?