Add GatewayParameters CRD for dynamic proxy configuration

[jenshu]

Description

Add support for customization of the dynamically provisioned proxy resources in Gloo Gateway. The customization is done via a new GatewayParameters CRD which enables configuring various aspects of the resources (deployment, service, etc) that are created by the GG deployer. A GatewayParameters is attached to a Gateway via an annotation gateway.gloo.solo.io/gateway-parameters-name on the Gateway, which points to the name of a GatewayParameters in the same namespace as the Gateway.

Example configuration of a Gateway and GatewayParameters is shown below (not valid values, just showing what all the configurable fields are):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http
  namespace: gloo-system
  annotations:
    gateway.gloo.solo.io/gateway-parameters-name: "my-gw-params"
spec:
  gatewayClassName: gloo-gateway
  listeners:
  - protocol: HTTP
    port: 8080
    name: http
    allowedRoutes:
      namespaces:
        from: All
---
apiVersion: gateway.gloo.solo.io/v1alpha1
kind: GatewayParameters
metadata:
  name: my-gw-params
  namespace: gloo-system
spec:
  kube:
    deployment:
      replicas: 3
    podTemplate:
      extraLabels:
        pod-label-key: pod-label-val
      extraAnnotations:
        pod-anno-key: pod-anno-val
      securityContext:
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
      imagePullSecrets:
      - name: my-secret
      nodeSelector:
        node-sel-key: node-sel-value
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: disktype
                operator: In
                values:
                - ssd
      tolerations:
      - key: "key1"
        operator: "Equal"
        value: "value1"
        effect: "NoSchedule"
    envoyContainer:
      bootstrap:
        logLevel: debug
        componentLogLevels:
          upstream: debug
          connection: trace
      image:
        registry: quay.io/solo-io-test-123
        repository: gloo-envoy-test-repo
        tag: 1.2.3
        pullPolicy: Always
      securityContext:
        allowPrivilegeEscalation: true
        privileged: true
        runAsGroup: 4444
      resources:
        requests:
          memory: "64Mi"
          cpu: "250m"
        limits:
          memory: "128Mi"
          cpu: "500m"
    service:
      type: ClusterIP
      clusterIP: None
      extraLabels:
        svc-label-key: svc-label-val
      extraAnnotations:
        svc-anno-key: svc-anno-val

Code changes

• Added the protos and codegen to generate the GatewayParameters CRD and Go types.
  • Note: GE classic already has code to generate go structs etc from protos using solo-kit. For GG, we are generally using skv2 and controller-runtime, not the solo-kit/snapshot framework, so decided to keep the (skv2-based) GG codegen separate from (solo-kit-based) GE codegen.
  • imported some k8s core and apimachinery protos, to avoid needing to redefine the structs in our protos
• The deployer now checks for a GatewayParameters annotation on the Gateway to get any custom config values, and merges those with the (previously used) default values.
• Updated the way we inject the envoy image values (repo and tag) from GlooEE. It is now done via a new GetEnvoyImage func (which returns different values in OSS vs EE) in K8sGatewayExtensions, which is passed to the GG deployer. Removed the temporary K8sGatewayExtensions workaround that was being used before.

Notes:

• Modifying a GatewayParameters object that is already being referenced by a Gateway will cause the deployer to re-run and update the proxy deployment.
• Adding/removing/modifying the params annotation on a Gateway will not in itself cause the deployer to re-run, since annotation changes don't trigger Gateway reconciliation. Once the Gateway parametersRef field is available and we move over to using that instead of the annotation, this issue should be resolved.

Testing:

• There are deployer unit tests covering many scenarios
• e2e enterprise tests are in progress and tracked separately

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

[howardjohn]

kubernetes-sigs/gateway-api#2924 has merged

[lgadban]

This looks really good!
Nothing too controversial in my eyes.
Just a couple of questions/comments; will do another quick pass tomorrow morning

[howardjohn]

Overall LGTM