Sustained CPU usage spikes when app out of sync

[davekonopka]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Whenever an ArgoCD managed app gets out of sync with an error that can't be resolved, like a secret that can't be accessed due to permissions errors, a few of our ArgoCD services have a sustained spike in CPU usage. As soon as the underlying issue is fixed outside of Argo and the app sync resolves the CPU usage falls back to normal levels. It's as if the services are working overtime. This happens frequently in our lower environments as new apps are being onboarded with ArgoCD. The CPU will stay sustained at the higher level for up to a day or more until the sync issue is resolved.

To Reproduce

1. Get an ArgoCD app into a state where it is out of sync and can't be resolved.

Expected behavior

ArgoCD flags an app out of sync but does not consume significantly higher CPU resources in multiple components.

Screenshots

Application controller pod cpu usage before and after app is out of sync.

Repo service pod cpu usage before and after app is out of sync.

Redis pod cpu usage before and after app is out of sync.

Server pod cpu usage before and after app is out of sync.

Version

v2.14.4+3d901f2

Logs

Paste any relevant application logs here.

[nitishfy]

Where are you getting these metrics from?

[davekonopka]

Our Kubernetes cluster via Datadog.

[davekonopka]

Another example of an issue causing this behavior is a failed rollout. A single app is marked "degraded" with rollout, analysisrun, and pods degraded and replicaset spinning.

[blakepettersson]

Potentially solved with #23047. Can you try with 2.14.12 to see if your issue is resolved?

[agaudreault]

@davekonopka Yeah, this was affecting the app when they had an error during the sync validation, but the app health was unknown and not out of sync. I will investigate more in details if you can reproduce on 2.14.12 :)

Also can you get the full kubernetes manifest (kubectl get app <app_name> -o yaml), wait a minute, then get it again and compare both output. I want to know what changed.

And last question, how is this app created? Is it managed by an appset, does it have progressive sync, is it managed by another app (apps of apps) ?

Thanks :)

[davekonopka]

@agaudreault I deployed 2.14.12 and did some testing. CPU spikes but doesn't seem to sustain like it had been. I think the update has greatly improved the CPU usage we were seeing. If we run into it again I will grab the manifests a few times. Thanks for your help.

The apps are created empty by application set. We use Helm charts and Rollouts to deploy application changes.