[NEW] Excessive connection attempts to failed nodes in clusterCron() cause CPU overhead

[sarthakaggarwal97]

The problem/use-case that the feature addresses

In clusterCron(), the function clusterNodeCronHandleReconnect() attempts to reconnect to nodes whose link == NULL on every iteration (10 times per second by default). This leads to repeated connection attempts to nodes that are unreachable or in a failed (PFAIL or FAIL) state.

When there are multiple failing nodes, this retry logic results in:

• High CPU usage due to frequent connConnect() calls.
• Excessive memory allocation/free churn from repeatedly creating and destroying clusterLink objects.

Profiles of the connected nodes:

---

Engine CPU / Used Memory metrics:

The initial spike is during the time when large numbers of primary nodes (n/2 - 1) were killed, and subsequent increase in the memory seems to be due to frequent reconnect attempts based on the above profile.

---

Once the failed nodes are resumed, the compute is back to its steady state levels.

Description of the feature

A backoff mechanism to avoid so many reconnects with the failed nodes.

[madolson]

I think a backoff mechanism seems OK. We might just also limit the number of clients we attempt to establish a connection to each cron. For example, @hpatro implemented logic to only accept one new TLs connection per second. We might do something similar here, only try to establish one new outbound connection per loop. It can't just be the first node, we might need to start randomly somewhere in the iteration process.

[sarthakaggarwal97]

@madolson how do you think we can come up with a maximum number of outbound connection per loop? I am wondering if backoff and limiting outbound connection to one might increase the time for a node to join the cluster. We can run experiments with this tho, to understand consequences on small and large clusters.

[sungming2]

what do you think about adding per node backoff and randomly selecting a few nodes to attempt reconnection each cron loop?
e.g., 2 primaries and 1 replica. Maybe we could prioritize primaries by assigning them more weight in the selection logic.

[hpatro]

what do you think about adding per node backoff and randomly selecting a few nodes to attempt reconnection each cron loop? e.g., 2 primaries and 1 replica. Maybe we could prioritize primaries by assigning them more weight in the selection logic.

I feel this idea complicates thing. When there are lot's of disconnected nodes it's quite possible we might have stale data. So, prioritizing some nodes over other might not have lot of value.

Like Madelyn suggested, we can slow down the no. of outgoing connection created per cycle.

@madolson how do you think we can come up with a maximum number of outbound connection per loop? I am wondering if backoff and limiting outbound connection to one might increase the time for a node to join the cluster. We can run experiments with this tho, to understand consequences on small and large clusters.

The clusterCron runs at 100ms frequency. We can do rough estimation. If we establish 10 new outgoing connection per cycle, for 100/1000/2000 connections, it would take around 1/10/20 seconds.

[madolson]

The clusterCron runs at 100ms frequency. We can do rough estimation. If we establish 10 new outgoing connection per cycle, for 100/1000/2000 connections, it would take around 1/10/20 seconds.

I feel like 10 seems Ok. The only bias we might try to introduce is when we have the inverse link. So if a node has a link established to us we connect to it without bounds, but if we don't we only do like 10 a second. That way when a node is isolated and comes back, it will establish all the inbound links and then immediately do all the outbound. When it's idle and disconnected, it won't try as hard.