Alternative way to include AWS EKS cluster

[tmisch]

We are hosting an Argo CD installation on a Kubernetes cluster (non AWS EKS) inside our company internal network. The cluster has access to the internet and we want to add an AWS EKS based cluster to Argo CD, to let Argo CD manage some infrastructural applications on the AWS EKS cluster that are required to deploy and run our own application e.g. for demonstration purposes outside the company.

The version of Argo CD currently running on our systems is v2.12.6.

I tried to add an EKS cluster to Argo CD using declarative setup as described in the official documentation Using An AWS Profile For Authentication.

The aws-auth config map is available in the EKS cluster and the corresponding IAM user already allows us to access the cluster and successfully deploy applications to it from within our deployment pipelines.

I have added the volume and volumeMount with the profiles file (resp. credentials file) from a secret to the deployment of the controller and the server component, and verified the mounted file in the two pods.

But i left out the aws_session_token from the credentials file, since session tokens provide temporary access (at least according to AWS_SESSION_TOKEN in AWS documentation ) and i think that permanent access is required for Argo CD. I do not want to update my secret credentials file in Kubernetes every hour or day!? In other use cases, it is sufficient to specify aws_access_key_id and aws_secret_access_key in the credentials file.

Unfortunately, using this configuration the cluster cannot be successfully integrated in Argo CD. Instead the following error is shown: error synchronizing cache state : Get "https://<...>.eks.amazonaws.com/version?timeout=32s": getting credentials: exec: executable argocd-k8s-auth failed with exit code 20 (Client.Timeout exceeded while awaiting headers)

I was wondering about the value of the profile configuration variable in the cluster secret mentioned in the docs, that points to the credentials file. As far as i know from the AWS documentation, the environment variable AWS_PROFILE specifies the name of a profile inside the credentials file, not the credentials file itself. By checking the sources of Argo CD and argocd-k8s-auth i came to the conclusion, that the meaning of the profile config setting could be identical to this environment variable. Due to that i changed configuration of the Argo CD helm chart and the cluster secret and was able to successfully integrate the cluster and deploy a test application to it using Argo CD.

These are the changes to my configuration:

• In the cluster secret, place the name of a profile that is listed in the credentials file instead of the path to the credentials file:

  apiVersion: v1
  kind: Secret
  metadata:
    name: mycluster-secret
    labels:
      argocd.argoproj.io/secret-type: cluster
  type: Opaque
  stringData:
    name: "mycluster.com"
    server: "https://mycluster.com"
    config: |
      {
        "awsAuthConfig": {
          "clusterName": "my-eks-cluster-name",
          "roleARN": "arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>",
          "profile": "default"
        },
        "tlsClientConfig": {
          "insecure": false,
          "caData": "<base64 encoded certificate>"
        }
      }

• Define AWS standard environment variable AWS_SHARED_CREDENTIALS_FILE using helm values for argocd-server and argocd-application-controller in addition to the volume and volumeMount specification already mentioned in Argo CD documentation:

  controller:
  env:
  - name: AWS_SHARED_CREDENTIALS_FILE
    value: /path/to/credentials/file
  
  volumeMounts:
  - name: aws-shared-credentials-file
    mountPath: /path/to/credentials/file
    subPath: credentials
    readOnly: true
  
  volumes:
  - name: aws-shared-credentials-file
    secret:
      secretName: aws-shared-credentials-file
  
  server:
    env:
    - name: AWS_SHARED_CREDENTIALS_FILE
      value: /path/to/credentials/file
  
    volumeMounts:
    - name: aws-shared-credentials-file
      mountPath: /path/to/credentials/file
      subPath: credentials
      readOnly: true
  
    volumes:
    - name: aws-shared-credentials-file
      secret:
        secretName: aws-shared-credentials-file

To my opinion, this setup is now very close to official AWS documentation.

Should the Argo CD documentation get updated to show this way of configuration in section Using An AWS Profile For Authentication?

BTW:
I needed to use yaml keys volume and volumeMounts instead of keys extraVolumes and extraVolumeMounts stated in the documentation. This could also be updated.

[avosepp]

I was linked here after running into a similar issue with the profile use in the cluster secret config. If you are correct, which I am going to attempt to validate. Then I would argue that the ArgoCD documentation on this issue is incorrect or misleading and an issue should be raised.

[avosepp]

@tmisch I think I'm pretty close to your solution but still running into the Auth timeout issue. I was wondering if you are on the CNCF Slack channel? If you're open to a few moments of time to walk through my test and see if I'm missing a step from yours? Thank you.

[tmisch]

@avo-sepp , i am not on that slack channel, yet, but may be i can join!? Can you send me a link?

To check whether the above configuration is basically working, i tried the following steps:

• open a shell into the argocd-server pod
• check environment variable AWS_SHARED_CREDENTIALS_FILE is set
• ensure that credentials file is mounted and available at the path pointed to by AWS_SHARED_CREDENTIALS_FILE
• manually run the argocd-k8s-auth util to see whether a session token can be obtained successfully, e.g.: argocd-k8s-auth aws --cluster-name my-eks-cluster
  optionally with the --profile argument, if the credentials file does not contain a default profile

[avosepp]

You can join using this link but I just found it on Google.

I was actually able to verify that your approach works! We've connected two clusters in two separate AWS accounts and tested it changes which profile it uses based on the credentials file.

I am for updating the ArgoCD documentation to reflect this. I believe your example here is the correct way to use this.

[JamesHawkinss]

+1 thank you so much for this! The ArgoCD documentation is flawed, especially with the incorrect Helm example and missing env variables. Legend!