Load between controllers (argocd-application-controller) is not evenly distributed

[vikas027]

Describe the bug

I have a ArgoCD High Availability setup where I have also scaled the number of replicas in argocd-application-controller as shown in the documentation.

To Reproduce

• Follow the steps to deploy ArgoCD in HA mode
• Edit the argocd-application-controller as below

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: argocd-application-controller
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: argocd-application-controller
        env:
        - name: ARGOCD_CONTROLLER_REPLICAS
          value: "3"

Expected behavior

I was expecting the controller to distribute the load to all three controllers but only one took up all the load, rest two are sitting idle.

Screenshots

All pods running in HA mode

❯ k get po                        
NAME                                      READY   STATUS    RESTARTS   AGE
argocd-application-controller-0           1/1     Running   0          160m
argocd-application-controller-1           1/1     Running   0          160m
argocd-application-controller-2           1/1     Running   0          161m
argocd-dex-server-7b6f9b7f-qh4kv          1/1     Running   0          3h6m
argocd-redis-ha-haproxy-d6dbf6695-4q5cj   1/1     Running   0          3h4m
argocd-redis-ha-haproxy-d6dbf6695-4sh7k   1/1     Running   0          3h5m
argocd-redis-ha-haproxy-d6dbf6695-hjn2d   1/1     Running   0          3h4m
argocd-redis-ha-server-0                  2/2     Running   0          176m
argocd-redis-ha-server-1                  2/2     Running   0          177m
argocd-redis-ha-server-2                  2/2     Running   0          179m
argocd-repo-server-5f4d4775d4-4mw4j       1/1     Running   0          173m
argocd-repo-server-5f4d4775d4-vhgxk       1/1     Running   0          174m
argocd-server-86896bd76f-gz48t            1/1     Running   0          173m
argocd-server-86896bd76f-k5r9h            1/1     Running   0          174m

Screenshot of the pods resources

Version

❯ argocd version                                                                                                                        
argocd: v2.0.1+33eaf11.dirty
  BuildDate: 2021-04-17T04:23:35Z
  GitCommit: 33eaf11e3abd8c761c726e815cbb4b6af7dcb030
  GitTreeState: dirty
  GoVersion: go1.16.3
  Compiler: gc
  Platform: darwin/amd64

[fewkso]

I think the load is distributed/sharded by cluster, so it might be more of an improvement than an actual bug
https://github.com/argoproj/argo-cd/blob/master/controller/appcontroller.go#L1517

[PatTheSilent]

Having an option to shard by something else than cluster would be much appreciated. Because I really dislike having single points of failure I have an ArgoCD stack in each of my clusters and due to this limitation I can only vertically scale the Application Controller, which is far from ideal.

[cwwarren]

Use case: we deploy a number of QA/Staging/Preview environments for one product in a single cluster to save costs. In addition to having many more environments (ArgoCD Applications) these environments are much higher churn, being updated anywhere from a few times a day to many times per hour at peak.

Ideally applications would be evenly distributed across shards to balance load so we aren't stuck over-allocating resources to underutilized shards.

Update 8/12 - ArgoCD 2.1.0-rc3 reduced resource use significantly, but the issue of highly imbalanced shards remains.

[maxbrunet]

I have written a script parsing the output of the argocd-util cluster stats command and suggesting a shard number for each cluster based on their resources count:

https://gist.github.com/maxbrunet/373374690b5064203e5e714de97d37fa

The script currently works offline, but we could imagine integrating such logic in the application controller or build a dedicated service around it. The result would likely read the cluster stats from Redis directly.

The algorithm is not perfect as I note in the caveats (see README.md), we could also base it on a different metric and/or work with a static number of controllers.

[viktor42b]

We have the same use case as well where we usually have 3 to 6k of ArgoCD applications in our staging environment. The fact that the sharding is on a per-cluster basis instead of per-app is not helping much because we deploy everything into the same cluster (in staging).

[kvendingoldo]

We have the same issue with load, only one shard is working and during the high load it fails, but all others do not pick tasks.

[aiceball]

Same scenario here, we have a very heavy application deployed on a single cluster and adding another application controller replica does not distribute the load evenly.

[dtelaroli]

I have the same issue.
With 3 replicas, 1 replica restarts each time when it drinks the entire instance memory (16Gb) and the 2 other use 2Gb of memory at most.
I will upgrade the node-group to 32Gb and to pray to Lord to be enough.

[musabmasood]

Same here. We have 8 application-controllers and 8 clusters managed by this ArgoCD instance. As you can see the usage is really not distributed.