NginxProxy service annotations (AWS load balancer controller)

[messiahUA]

NginxProxy spec doesn't support setting annotations for the service, and I'm using patches, but I have noticed multiple times a problem related to the aws load balancer controller. I think this is a race condition.

When I create a new NginxProxy I see that the actual load balancer in AWS doesn't have any configuration set in the annotations, so I think the problem is that first the service is being created, then aws controller processes it, and only then the patch is applied, and this causes a broken configuration that can be fixed only by re-creation of the load balancer.

I think annotations should be supported natively and must be set initially without patches.

[bjee19]

Hey @messiahUA, thanks for opening this discussion/issue. I wanted to confirm, when you refer to the service, which service are you referring to?

Currently with NGF, when you deploy NGF and a Gateway (NGINX Pod), two types of services are spun up. One service for NGF which doesn't interact with traffic and is simply used for internal communication between NGF and the data plane pods, and another service which fronts the NGINX Data plane pod.

If you are referring to the latter, annotations can be set through the gateway's infrastructure field. Here are some instructions: #4822 (comment). See here for more details on the infrastructure field: https://gateway-api.sigs.k8s.io/guides/user-guides/infrastructure/.

If you are talking about the former (service fronting NGF), if you could post your NginxProxy config, that would be helpful.

[messiahUA]

I refer to the front service of the data pods. I have looked only at the nginxproxy spec, so this is how it is looking for me (I have omitted all irrelevant fields):

apiVersion: gateway.nginx.org/v1alpha2
kind: NginxProxy
spec:
  kubernetes:
    service:
      patches:
      - type: StrategicMerge
        value:
          metadata:
            annotations:
              service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
              service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
              service.beta.kubernetes.io/aws-load-balancer-name: automation
              service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
              service.beta.kubernetes.io/aws-load-balancer-scheme: internal
              service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
              service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
              service.beta.kubernetes.io/aws-load-balancer-type: external

Didn't check the Gateway. Now I see it has infra annotations, so yes, it looks like the correct place to place them. Thank you.

[messiahUA]

Just a note for anyone reading this discussion: as of right now, the latest stable gateway api crd has "maxProperties" set to 8 for annotations, which is very limiting. There was already a change to increase this to 16, which is better, but still, there is a chance to hit the limit because there are many more possible annotations...