Nebari Traefik ingress and AWS load balancers?

[mwengren]

I'm trying to understand Nebari/Traefik's connection to the load balancer created in AWS when Nebari is successfully deployed. I'm deploying Nebari to a pre-existing AWS VPC with a public/private subnet architecture that looks like this presently:

     AWS us-east-2 AZ 1                                  AWS us-east-2 AZ 2

     public subnet                                        public subnet
     [xyz.xyz.xyz.xyz/27]                              [xyz.xyz.xyz.xyz/28]

     private subnet                                       private subnet
     [10.xyz.xyz.xyz/23]                                  [10.xyz.xyz.xyz/23]

When the Nebari terraform deployment completes, an AWS Classic load balancer is created across both AZs (assuming I pass it a subnet from each of the above AZs in my nebari-config.yaml). I also need to have an AWS internal load balancer with no public endpoints due to my networking set up and requirements to use private networking for all AWS egress.

Currently, I'm using these annotations to do that:

ingress:
  terraform_overrides:
    load-balancer-annotations:
      "service.beta.kubernetes.io/aws-load-balancer-subnets": "subnet-..., subnet-..."
      "service.beta.kubernetes.io/aws-load-balancer-internal": "true"

My main question is: is a Classic load balancer always created by the Traefik k8s LoadBalancer config in Nebari? Or are other types of AWS ELBs sometimes created in different circumstances than mine?

In my case the Classic LB is configured with listener protocols/ports that match the above config in the terraform code (80, 443, 8022, etc), but with corresponding instance protocols/ports that seem arbitrary - or at least not specified in the terraform. I can see in the nebari-traefik-ingress k8s service how the node port settings match the instance ports in the Classic LB, so I think I understand the connection there and more or less how the traffic is routed.

It isn't clear to me:

• How the instance/node ports are set, and whether or not they can be modified
• Related to that, is it possible to override the Classic LB that Nebari/Traefik creates to use instead a network load balancer (NLB) or application load balancer (ALB) if needed? Or, instead, if the load balancer created via the Nebari terraform deploy is somehow managed by Traefik and shouldn't be touched in order to work properly?

Is there any documentation about managing Nebari-connected load balancers that I might have missed? These two pages are the relevant docs I've found: enhanced security and ingress overrides.

Thx!

[mwengren]

Answering my own question here to close this out and in case it helps someone else.

From my testing, it was clear that Nebari/Traefik will only create an AWS Classic Load Balancer, and I was not successful in either 1) deploying an alternate LB type (such as an ALB or NLB) via nebari deploy or 2) using an alternate LB type that I created manually via AWS Console. The Classic LB is the only option.

I also read in the Traefik documentation (I believe, it was awhile ago) that the Traefik NodePorts (or possibly Instance ports) I believe that correspond to the entryPoints specified in the Nebari Traefik config are arbitrarily assigned (in my case in the 30000 range) - lost the doc link for that unfortunately.

The good news is with some changes I was able to make the Classic LB work for my unorthodox AWS networking set up with a few specific settings in nebari-config.yaml. See: https://github.com/orgs/nebari-dev/discussions/3123 for info about that aspect.