Support Large Valkey Cluster

[hpatro]

We aim to scale Valkey to support large deployments (2000 nodes) in cluster-enabled setup with the following criteria:

• Failure detection period: cluster-node-timeout - 15 seconds

• Deployment strategy

  • Nodes spread across 3 or more isolated data center zones (availability zones)
    • 1 primary / 1 replica per shard i.e. 1000 primary / 1000 replica(s)
    • 1 primary / 2 replica per shard i.e. 666 primary / 1332 replica(s)

• Resilience against simultaneous failure scenarios:

  • 33% of overall node failures

Expectations:

• Steady and predictable CPU utilization under normal and failure conditions.

• Consistent and bounded failure detection time (~15 seconds).

• Predictable failover time bounded to minimal downtime.

• Convergence (cluster state stabilization) within bounded limits post-failure recovery.

Alternatives to Consider:

• Client-side sharding across multiple smaller clusters.

• Using a proxy solution to route commands/data across multiple smaller clusters.

We will be using this issue as the high level tracker. Will be posting updates here periodically as well about the benchmark results.

High Level Areas:

Cluster initialization / setup:

• Accept limited number of new cluster link connections per cycle #2009

Concurrent node failure detection / failover:

• Logs consuming significant compute during cluster node failure detection #2076 / Avoid log spam about cluster node failure detection by each primary #2010
• [NEW] Excessive connection attempts to failed nodes in clusterCron() cause CPU overhead #2122
• [NEW] Performance bottleneck in clusterNodeCleanupFailureReports under heavy failure load #2139
• Cluster fails to recover when ~50 % of primaries are killed (majority still alive) #2181
• Mark primary node as alive immediately if reachable and failover is not possible #1927
• [NEW] Propagate node availability-zone information across cluster and make failover smarter #1937
• [NEW] Faster cluster failover #2023

Convergence / Information dissemination:

• [NEW] Measure cluster convergence time #1924

Observability:

• Add node pfail and fail count to cluster info metrics #1910
• [NEW] Measure the amount of network in/out over cluster bus #1929
• Log failed cluster node(s) state periodically to capture transient state for debuggability #2011
• Introduce CLUSTERLOG command support #2029

Steady state [CPU utilization]:

• Make gossip message size configurable and deterministic #2291
• Reduce steady state CPU utilization for clusterbus via SWIM protocol #2369

[hwware]

My question is how to deploy the testing environment for this?

[hpatro]

My question is how to deploy the testing environment for this?

Good question. This was one of the area we wanted to invest on and we're currently working on that. The idea is to create tooling to create Valkey cluster on VMs, run various test/benchmarking scenarios and capture datapoints/metrics. The tooling is WIP and we plan to open-source it soon after a bit of sanitization.
With the tooling, we can run desired scenarios before each release to certify there has been no regression in the behavior.

High level overview of the tool (in current state):

• Uses Amazon EC2 instances
• Sets up networking amongst the nodes
• Creates Valkey cluster
• supports impairment of Valkey process
• Captures various metrics from INFO/CLUSTER INFO/system
• Publish to CloudWatch.

This should be extendable to other infrastructure provider as well.

Note: It's quite expensive to run it for extended period, so one needs to be mindful of that.

[zuiderkwast]

#2122 and #2139 optimize the cluster when some node is failing. I don't see any issue or PR to optimize the cluster traffic in the steady state when no node is failing. @hpatro you've mentioned SWIM earlier, that we can take some inspiration from that to reduce cluster ping-pongs. If you have an issue for this, let's add it to this list?

Any other ideas to reduce overhead in steady state, such as changing some linked lists to arrays?

I think we should not focus only on the default 15 seconds node timeout. It is too high for some setups. If a system requires maximum 5 seconds from when a node starts to fail until a failover is completed, we need to set node timeout to 2000. If we fix #2023 (remove or reduce the fixed 500ms + random 0-500ms delay before failover starts) it let's us increase node timeout from 2000 to 3000 and this in turn reduces cluster traffic a lot so we can use larger clusters. So let's add #2023 to this list?

An idea regarding testing: As a complement to testing with a very large cluster, we could set node timeout very low to increase the cluster bus traffic in a smaller cluster. This is not the same (of course) but in some aspects it's similar. We can analyze the similarities and use it when possible.

[hpatro]

#2122 and #2139 optimize the cluster when some node is failing. I don't see any issue or PR to optimize the cluster traffic in the steady state when no node is failing. @hpatro you've mentioned SWIM earlier, that we can take some inspiration from that to reduce cluster ping-pongs. If you have an issue for this, let's add it to this list?

Was planning to submit with a POC but can submit an issue as well to discuss more.

Any other ideas to reduce overhead in steady state, such as changing some linked lists to arrays?

One potentially easy win is reducing the packet size (configuring gossip size) was another idea which has been brought up quite a few times. The challenge with that, it will change characteristic of failure discovery and information dissemination. So, needs to be well tested and documented as well.

I think we should not focus only on the default 15 seconds node timeout. It is too high for some setups. If a system requires maximum 5 seconds from when a node starts to fail until a failover is completed, we need to set node timeout to 2000. If we fix #2023 (remove or reduce the fixed 500ms + random 0-500ms delay before failover starts) it let's us increase node timeout from 2000 to 3000 and this in turn reduces cluster traffic a lot so we can use larger clusters. So let's add #2023 to this list?

That's fair. Wanted to have a baseline with default timeout and reduce the steady state CPU utilization further. Let me add this as well.

An idea regarding testing: As a complement to testing with a very large cluster, we could set node timeout very low to increase the cluster bus traffic in a smaller cluster. This is not the same (of course) but in some aspects it's similar. We can analyze the similarities and use it when possible.

+1. This is a good way to measure the CPU utilization w.r.t. message volume.

[zuiderkwast]

• [NEW] Migrate non topology update cluster bus messages to light header type #932

[hpatro]

#2122 and #2139 optimize the cluster when some node is failing. I don't see any issue or PR to optimize the cluster traffic in the steady state when no node is failing. @hpatro you've mentioned SWIM earlier, that we can take some inspiration from that to reduce cluster ping-pongs. If you have an issue for this, let's add it to this list?

Was planning to submit with a POC but can submit an issue as well to discuss more.

Created an issue for integrating SWIM protocol to cluster for failure detection and reduce steady state CPU utilization #2369. Folks interested in it please share your thoughts.