High Application Controller CPU/Memory Usage in version 3.0.11

[HarleyB123]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

For context - we are running a pretty large (~30k applications) setup with ArgoCD. We are running on GKE, with application controller shards managing clusters across all three cloud providers.

For sharding, we are using legacy, but with some custom logic. We have a controller that sets ARGOCD_CONTROLLER_REPLICAS to be math.MaxInt32, and the ARGOCD_CONTROLLER_SHARD so that each application controller manages at most 10 clusters.

When upgrading to version 3.0.11, we followed the docs and accepted all the new changes, except for ignoreResourceStatusField, which we kept as crd. (this item in the upgrade docs).

When updating from version 2.14.3 to 3.0.11, we noticed a significant increase in CPU utilisation across all of our application controllers.

We profiled CPU time of all application controllers, here is a 200 second CPU profile:

From this, we decided to change the ARGO_CD_UPDATE_CLUSTER_INFO_TIMEOUT setting to 59 seconds (max is 1 minute), which is why you see a dip in CPU utilisation in Grafana - but it is nowhere near where it previously was before the upgrade.

As well as this, we also noticed that some shards were OOM'ing when they previously were not. Investigations into these shards revealed that the clusters being managed by these shards have significantly more resources to watch (they were Azure clusters, which have azure-service-operator custom resources) than the others. However, even with reducing the number of resources watched via exclusions, we are still seeing OOMs - albeit less frequently.

Whilst investigating the issue, we took repeated heap dumps of the application controller that periodically OOM'ed, here is the heap dump 15 seconds before the application controller OOM'ed:

When enabling the debug logs, we noticed that OOM's pretty much as soon as it needed to reconcile a resource (or sometimes even earlier , see below for a familiar log pattern):

Checking if cluster x with clusterShard y should be processed by shard z
Checking if cluster x with clusterShard y should be processed by shard z
Checking if cluster x with clusterShard y should be processed by shard z
Checking if cluster x with clusterShard y should be processed by shard z

To Reproduce

Difficult to reproduce, hence providing as much context about our setup as possible!

Expected behavior

Similar (or better) performance from ArgoCD when upgrading.

Version

argocd: v3.0.11+240a183
  BuildDate: 2025-07-10T14:53:50Z
  GitCommit: 240a1833c0f3ce73078575d692146673e69b6990
  GitTreeState: clean
  GoVersion: go1.24.4
  Compiler: gc
  Platform: linux/arm64

[crenshaw-dev]

The OpenAPI stuff looks pretty darn heavy.

I wonder if it would be possible to pull a heap profile during startup in 2.14 and see if it helps indicate the difference.

[crenshaw-dev]

I see a lot of stuff under GetDestinationCluster. I think 3.0 was the version where we swapped out logic that populated the spec.destination.server field in the Application once in favor of clearer logic that pulled the destination cluster whenever needed.

If we're calling that function a lot more, and if that function is being slow for some reason, that could explain some of the problem.

[crenshaw-dev]

Looks like a ton of time is being spent parsing TLS keys from cluster secrets.

[HarleyB123]

I see a lot of stuff under GetDestinationCluster. I think 3.0 was the version where we swapped out logic that populated the spec.destination.server field in the Application once in favor of clearer logic that pulled the destination cluster whenever needed.

If we're calling that function a lot more, and if that function is being slow for some reason, that could explain some of the problem.

Do you have a link to this change? It sounds potentially suspect. Not really comparable but our significantly smaller staging instance of ArgoCD in terms of both clusters manager (20 clusters and 600 applications) had less CPU utilisation on the application controllers when upgraded.

When you saw "whenever needed", what conditions would ArgoCD want to populate/repopulate this?

I wonder if it would be possible to pull a heap profile during startup in 2.14 and see if it helps indicate the difference.

I think this will be tricky to get for production, is it safe to downgrade ArgoCD like this?

[crenshaw-dev]

This is the change: #21189

There are a lot of places in code that require destination cluster info. The CPU profile chart doesn't seem to indicate any particular code path as the biggest offender.

Should be safe to downgrade.

[crenshaw-dev]

Looks like maybe canProcessApp is a significant offender, but not the majority caller.

[HarleyB123]

Nice, we will look into downgrading to 2.14 next week and I'll provide the CPU and Heap Dump profiles once done.

Realised I forgot to mention in my initial report, production ArgoCD for us is managing close to 1k clusters - so I think it makes sense that GetDestinationCluster is a culprit.

Does each application controller call GetDestinationCluster for just the clusters it manages? Or for every cluster that the system is aware of?

[rumstead]

related PR #24080 and issue #17277

@mtbennett-godaddy as an FYI

[FourFifthsCode]

@HarleyB123 we are currently investigating this and testing a possible fix

[PascalVA]

We recently upgraded ArgoCD from v2.14 to v3.0 and have also noticed an increase in CPU as well:

We manage (only) 25 clusters but 2000 applications so, perhaps it is worth looking in the the reconciliation process

(the query aggregates the cpu seconds of our 3 argocd application controler pods)

EDIT: We do not have the argocd-server-tls secret on our cluster and are not seeing these messages.

[mtbennett-godaddy]

@PascalVA Even if you don’t use argocd-server-tls, tls.X509KeyPair will still be called on the self-signed cert in argocd-secret (unless you have insecure mode enabled).

#24080 caches the cert (whether external or self-signed) to avoid calling tls.X509KeyPair every time GetSettings is called.