[NEW] REPLICAOF NO ONE using a coordinated failover

[KarthikSubbarao]

The problem/use-case that the feature addresses

Currently, when we execute REPLICAOF NO ONE on a node in a cluster mode disabled setting, it turns into a primary. However, the node might not be up to date and can be lagging behind in comparison to other replica nodes. This can lead to partial data loss as well as causes replica nodes with higher repl offsets to full sync with the new primary.

Description of the feature

I was hoping to get input on the possibility of extending REPLICAOF NO ONE to use a coordinated failover approach (similar to the FAILOVER command) of pausing writes on the primary, allowing the chosen node (role of replica) to sync fully with the current primary, demote primary, and PSYNC failover on the chosen node to promote it to a primary.

This will limit data loss and unavailability of the cluster during the failover when using the REPLICAOF NO ONE.

The primary use case is for systems that are using the REPLICAOF NO ONE command to issue failovers on clusters in a cluster mode disabled setting.

Alternatives you've considered

Upgrading systems to use the FAILOVER command as it uses the coordinated failover approach

[madolson]

The historical context was that in Redis we thought that it made more sense to send the command to the primary to basically execute a "step down" command, and it would pick the most up to date replica. There is no technical reason why we couldn't support both a strategy to do the failover from the replica or the primary.

Today in cluster mode failovers are started on the replica, but there are cases where it would be useful to have it trigger on the primary. For example, if a container is notified it is going to be deleted, it makes sense for the primary to try to elect another replica.

I would be supportive of both. Being able to send a command to a replica to "safely promote itself" as well as "having a primary safely demote itself". Would love input from @PingXie @soloestoy @enjoy-binbin

[zuiderkwast]

For example, if a container is notified it is going to be deleted, it makes sense for the primary to try to elect another replica.

This is exactly what the new trigger-failover-on-primary-shutdown feature does: #1091

[PingXie]

Good point, @KarthikSubbarao. this is a real data safety problem for standalone setups.

I'm not a fan of changing the semantics of REPLICAOF NO ONE, though. that command is a simple primitive; it does one thing, and adding coordinated failover logic to it feels out of place and could break things for users who expect the current behavior.

Instead, what if we expand the existing FAILOVER command to allow it to run on a replica?

This solves the problem cleanly by using a command already built for this purpose. it would also bring direct parity with CLUSTER FAILOVER, which is initiated from a replica in the same way. this makes the behavior more consistent across both modes.

also agreed with @zuiderkwast's point on the primary-initiated shutdown behavior (#1091). it is great to align with the cluster mode in the graceful shutdown path.

[zuiderkwast]

I'm not a fan of changing the semantics of REPLICAOF NO ONE, though. that command is a simple primitive

I agree with this. I even think it can be a breaking change. REPLICAOF NO ONE means decouple the nodes into two individual primaries.

Instead, what if we expand the existing FAILOVER command to allow it to run on a replica?

What syntax do you suggest? The current syntax is

FAILOVER [TO host port [FORCE]] [ABORT] [TIMEOUT milliseconds]

How can we extend this to be sent to a replica without too much confusion?

[madolson]

This is exactly what the new trigger-failover-on-primary-shutdown feature does: #1091

This is only a partial solution. There are other ways a system might try to step down.

[madolson]

I agree with this. I even think it can be a breaking change. REPLICAOF NO ONE means decouple the nodes into two individual primaries.

Why don't we just support an optional argument? REPLICAOF NO ONE COORDINATED ? I don't really see why we should attach the semantics to the FAILOVER command?

[zuiderkwast]

REPLICAOF NO ONE is not for manual failover. It's for detaching a replica without affecting the primary. It stops what REPLICAOF host port started. Giving it coordinated failover semantics is bad API design IMHO.

[madolson]

REPLICAOF NO ONE is not for manual failover. It's for detaching a replica without affecting the primary. It stops what REPLICAOF host port started. Giving it coordinated failover semantics is bad API design IMHO.

I think it's clearer than sending FAILOVER to the replica, a command designed to be sent to the primary to failover to some replica. Do we want a third API is warranted here then? The alternative is we just tell users they need to send the command to the primary and not change anything.

[zuiderkwast]

The alternative is we just tell users they need to send the command to the primary and not change anything.

Yeah! I like this. @KarthikSubbarao you even mentioned this under Alternatives considered. Is there any problem with this?

[madolson]

If we want to stick with failover, maybe FAILOVER TAKEOVERFROM IP PORT?

[PingXie]

If we want to stick with failover, maybe FAILOVER TAKEOVERFROM IP PORT?

The replica should know its primary. I would think we can match CLUSTER FAILOVER by FAILOVER [FORCE | TAKEOVER].

[soloestoy]

The alternative is we just tell users they need to send the command to the primary and not change anything.

Yeah! I like this. @KarthikSubbarao you even mentioned this under Alternatives considered. Is there any problem with this?

I like this too, it's not difficult to send FAILOVER [TO host port [FORCE]] on primary IMHO.

[KarthikSubbarao]

Yeah! I like this. @KarthikSubbarao you even mentioned this under Alternatives considered. Is there any problem with this?

Thank you for the feedback on this issue.

Yes, I listed the FAILOVER command as an alternative.

However, the one issue with this command (as mentioned earlier in this thread) is that the CLUSTER FAILOVER command is executed on the replica that we want to promote via a coordinated failover

The FAILOVER command (which provides a coordinated failover) does not follow this semantic (above) as it requires the command to executed on the current primary. So, systems managing Valkey clusters today will have to use a slightly different logic if they want to implement coordinated failovers in CMD vs CME.

The options at a high level I thought were:

1. REPLICAOF NO ONE supporting a coordinated failover. However, it sounds like we want keep this command as it is today to not be a manual failover and simply detach itself. So, we can drop this option.
2. FAILOVER supporting being executed on replica nodes which we want to promote to primary. FAILOVER FORCE/TAKEOVER.
3. Not changing any Valkey commands, and having slightly different logic in how we manage our clusters for coordinated failovers for CMD vs CME

Going with (3) is not bad, but I was curious why the semantics were CLUSTER FAILOVER and FAILOVER are different and if this is a gap we want to address (with (2))