Improve sync performance in gitops-engine

[reggie-k]

After a discussion about Application Sharding at the contributors meeting, @crenshaw-dev suggested a much less complex and more immediate performance improvement, that can be achieved by the following in gitops-engine:

1. Using protobuf instead of json serialization
2. Performing cluster initialization using watch API instead of list

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

To Reproduce

Expected behavior

Screenshots

Version

Paste the output from `argocd version` here.

Logs

Paste any relevant application logs here.

[bygui86]

Hi guys
Thanks for opening this issue.
It's always good to improve performances, but I'm not sure to understand how improving sync performance can help solving the issue many users are facing since 4 years "issue distributing evenly the load between application controllers, based on apps instead on cluster".
@reggie-k can you please explain this a bit more?
Thanks a lot!

[crenshaw-dev]

The main (only?) benefit of sharding is to avoid hitting max node size. Sharding for any other reason just spreads the exact same load, plus some overhead, across multiple pods.

"Even load" isn't valuable aside from squeezing work into limited-size nodes.

We could spend a bunch of time trying to balance load across shards to avoid spiking through node limits, or we could focus on dropping the aggregate load by improving performance, thus allowing us to fit the (still uneven) load into available nodes or to avoid sharding altogether.

I think there are enough low-hanging performance improvements available to justify focusing on that instead of sharding.

[crenshaw-dev]

We're also simply running out of clever ways to even out load. The Application controller's main unit of load isn't an application, it's a cluster (due to the necessity of maintaining a cluster cache). Even if there's some way to avoid that architecture (and I'm not sure there is), the solution is likely more costly than readily-available performance improvements.

Sharding at the app level would be a bit like trying to split an airplane into multiple pieces to spread out the fuel consumption.

[bygui86]

Thanks @crenshaw-dev for your explanation.

We're also simply running out of clever ways to even out load. The Application controller's main unit of load isn't an application, it's a cluster (due to the necessity of maintaining a cluster cache).

I understand, the main reason for current architecture is cluster cache.

Even if there's some way to avoid that architecture (and I'm not sure there is), the solution is likely more costly than readily-available performance improvements.

I trust you maintainers on this and thanks for your effort. Performance improvements are always a good thing and absolutely very. But I'm not sure to understand why a hidden different solution (or even architecture) would be more costly?

The main (only?) benefit of sharding is to avoid hitting max node size. Sharding for any other reason just spreads the exact same load, plus some overhead, across multiple pods.

I don't fully agree on this. Let's consider a simple example: I have 2 clusters, the first with 10 apps, the second with 100 apps and ArgoCD with HA enabled. With cluster sharding, the load on the 2 app-controllers is going to be anyway unfair: the app-ctrl managing the second cluster has 10x the load in respect to the first one. Instead with app sharding the load should be around 55 apps for each controller. Or do I miss something?

"Even load" isn't valuable aside from squeezing work into limited-size nodes.

I agree here as well. But being forced to have some app-controllers with 1 cpu and some other with 10 cpus sounds not reasonable as well. We can't use limited-size nodes for everything, but we can't also waste resources.

We could spend a bunch of time trying to balance load across shards to avoid spiking through node limits, or we could focus on dropping the aggregate load by improving performance, thus allowing us to fit the (still uneven) load into available nodes or to avoid sharding altogether.
I think there are enough low-hanging performance improvements available to justify focusing on that instead of sharding.

I think that the effort to reduce load and improve overall performances is very welcome and appreciated. But as the topic of "app-based sharding" is around in the community since long time, I think that a deeper investigation on an alternative architecture could be useful.

[crenshaw-dev]

But I'm not sure to understand why a hidden different solution (or even architecture) would be more costly?

I'm only guessing, since I'm not aware of a proposed architecture which avoids the cluster-level cache granularity. But any re-architecture would be costly.

I have 2 clusters, the first with 10 apps, the second with 100 apps and ArgoCD with HA enabled. With cluster sharding, the load on the 2 app-controllers is going to be anyway unfair: the app-ctrl managing the second cluster has 10x the load

That's not accurate. App controller resource consumption is based on a lot of variables. Number of apps is maybe one of the least relevant variables. Number of resources monitored and resource churn are much more relevant, and those numbers are completely decoupled from app count.

the load on the 2 app-controllers is going to be anyway unfair... to have some app-controllers with 1 cpu and some other with 10 cpus sounds not reasonable

Fairness and reasonableness in load distribution aren't valuable by themselves.

we can't also waste resources

Using a single shard (assuming you have a node big enough to accommodate it) would avoid wasting resources.

I think that a deeper investigation on an alternative architecture could be useful.

I think time spent on that investigation would be better spent on cluster cache performance improvements. But I'd welcome a proposal.

[zimmertr]

The main (only?) benefit of sharding is to avoid hitting max node size. Sharding for any other reason just spreads the exact same load, plus some overhead, across multiple pods.

One other benefit is reliability. If you manage X clusters across Y shards (assuming they're evenly distributed) if shard 1 goes down application reconciliation should not be disrupted across Y-1 shards. Ensuring the reconciliation still occurs across most of the clusters.

To say it plainly, having one Application Controller is a single point of failure.

[crenshaw-dev]

That's fair. It's not redundancy, but it does limit the blast radius.

[bygui86]

@crenshaw-dev thanks for your answer and further explanation.

Is there a document listing all variables influencing app-controller resource consumption?

Using a single shard (assuming you have a node big enough to accommodate it) would avoid wasting resources.

I strongly agree with @zimmertr, a single shard is not a solution for some reasons like single point of failure, but also requiring a node big enough to accommodate it which is potentially really different from all other nodes (dedicated node pool only for app-controller?). And in general it's not a wise use of resources (and money!) for example if in the use case we have to spin up and tear down clusters depending on different factors.

I think time spent on that investigation would be better spent on cluster cache performance improvements.

I fully trust you on this, my knowledge of ArgoCD code base is not so deep.

But I'd welcome a proposal.

Thanks a lot for your effort!

[crenshaw-dev]

Is there a document listing all variables influencing app-controller resource consumption?

Not really, but most of them would be here: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

a single shard is not a solution

Fair, and it's possible I'm overstating the degree to which we can rely on a single-shard mode. But I think if we focus on controller performance, we'll make it a more viable option for more and more users while also improving things for folks who rely on multiple shards.