ApplicationSet with Environment Variables: Dynamic Cluster Identification and Config Management

[grok-rs]

I am currently managing several Kubernetes clusters with ArgoCD. Each cluster (e.g., develop-01, production-01, production-02) has its own ArgoCD instance. We maintain a GitOps repository that is structured like this:

  develop-01/
    kustomization.yaml
    grafana/
      overrides.yaml
  production-10/
    kustomization.yaml
    grafana/
      overrides.yaml
  shared/
    grafana/
      kustomization.yaml
      grafana.yaml

I want to achieve the following:

1. Shared ApplicationSet Configuration:

   • Define default values for applications in a shared configuration.
   • Avoid code duplication across multiple cluster-specific configurations.

2. Cluster-Specific Overrides:

   • Allow each cluster to override specific values (e.g., ingress host, resource limits) without repeating the entire configuration.

3. Simplified GitOps Workflow:

   • Each cluster should use its own kustomization.yaml file that references shared configurations (e.g., ../shared/grafana.yaml).
   • The ApplicationSet should dynamically apply the correct values based on the environment (cluster).

Current Challenge

Currently, I’m facing issues with duplicate configuration and manual effort when managing applications across multiple clusters. Here’s an example ApplicationSet YAML for a Grafana Helm chart:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: grafana-appset
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: [email protected]:repo-level/some-gitops-repo.git
        revision: HEAD
        files:
          - path: '*/grafana/overrides.yaml' # Here * relays on specific cluster
         # I want to use something like this
         # - path: '${environment}/grafana/overrides.yaml' 
         # where environment is name of cluster where argo is located

  template:
    metadata:
      name: grafana
    spec:
      project: default
      source:
        repoURL: https://grafana.github.io/helm-charts
        chart: grafana
        targetRevision: "8.6.4"
        helm:
          values: |
            ingress:
              enabled: true
              hosts:
                - {{host}}
      destination:
        server: https://kubernetes.default.svc
        namespace: grafana
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

References:
I see there is only limited set of default variables
https://argo-cd.readthedocs.io/en/latest/user-guide/environment-variables/

Also, will I be able to provide a custom OS environment with CMP?
https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/

So, the main question is how to restrict argocd to use only overrides from specific folder correlated to it's cluster?

[grok-rs]

Solved with https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Matrix/ and https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Cluster/

[martijnhybrit]

how did you solve it?
It's not clear from the docs how to leverage the generators for each ArgoCD instance. Would like to know what the config should be.

[grok-rs]

@martijnhybrit

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test-service
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - merge:
        mergeKeys:
          - name
        generators:
          - clusters: {}
          - list:
              elements:
                - name: dev-01
                  namespace: be01-dev
                - name: prod-02
                  namespace: be02-prod
                - name: prod-03
                  namespace: be03-prod
  template:
    metadata:
      name: 'test-service-{{ .name }}'
    spec:
      project: backend
      sources:
        - repoURL: 'https://gitlab.com/some_helmcharts_repo.git'
          path: test-service
          targetRevision: main
          helm:
            values: |
              service:
                enabled: true

              envVariables:
                ENVIRONMENT: "{{ .name }}"

              config:
                environment: "{{ .name }}"
                natsUrl: "nats://{{ .name }}-nats-cluster.{{ .namespace }}.svc.cluster.local:4222"

            valueFiles:
              - $values/{{ .name }}/apps/test-service.yaml
        - repoURL: 'https://gitlab.com/some-gitops-applications.git'
          targetRevision: main
          ref: values
      destination:
        server: '{{ .server }}'
        namespace: '{{ .namespace }}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
        - CreateNamespace=true

[grok-rs]

just values you can provide in file, in applicationSet variables which shared between clusters

valueFiles:
              - $values/{{ .name }}/apps/test-service.yaml

[martijnhybrit]

Thanks! that was what I was looking for