[NEW] Rolling downgrade, forward compatibility

[zuiderkwast]

The problem/use-case that the feature addresses

After a rolling upgrade of a cluster, if some problem is found, such as CPU or memory usage or some bug, the administrator wants to be able to a rolling downgrade again and take some time to investigate the problem.

When adding new nodes to a cluster, the standard procedure is to add new nodes as replicas to existing nodes. For replication to work, we require that a replica's RDB version >= primary's RDB version. This works for rolling upgrade, but it doesn't work for downgrading if the RDB version has changed.

Some system that I've been made aware of is stuck on Redis 6 (RDB 9) because of the requirement to be able to do rolling downgrades.

The cluster bus is already backward and forward compatible AFAIC. It would be good to avoid breaking that. :)

Note that we're only talking about adding new nodes of an old version to a cluster. We don't need to start an old node with a config file from a new version such as nodes.conf, nor do we want to reuse a node-id when replacing nodes in the cluster.

Description of the feature

Possibility to generate RDB in an older version than the latest, if no keys require the newer RDB version.

An example is the bump from RDB 10 to 11 (that I'm guilty of by introducing listpack encoding for sets). If we configure the node to generate RDB 10, the node can easily be made to store this listpack as a RDB_TYPE_SET (which is just each element as a string) instead of RDB_TYPE_SET_LISTPACK (a listpack dump). The bump from 9 to 10 is similar.

Alternatives you've considered

#59 = Full sync without RDB preamble, although it could be combined with a REPLCONF to check if the replica supports the current RDB version and fallback to AOF without preamble. (RDB is a bit faster and smaller in size.)

Additional information

Redis never supported downgrades. This would be a Valkey feature that would make Valkey superior in terms of compatibility.

Valkey/Redis	RDB	Why bump
Redis 5.0	9
Redis 6.0	9
Redis 6.2	9
Redis 7.0	10	Change ziplist to listpack for hashes, sets, etc.
Valkey 7.2	11	Add listpack encoding to sets
Valkey 8.0	11

[madolson]

The major decisions:

1. Do we want to be able to support this downgrade.
2. How far back to we want to support restoring into older versions? (At least one version, maybe much further)
3. How will we document this.
4. What is the format we will use to support forward compatibility. One option is to have the primary send an older RDB compatible object. Another option is to send it in AOF format.

Consensus from core meeting seems to say yes for 1.

[madolson]

We also want to create a separate issue outlining how we can add guardrails to prevent usage of new features which would break forward compatibility. The discussed example was for hash field expiration, which must necessarily change the RDB format.

[zuiderkwast]

A pre-condition we seemed to agree on is that a downgrade is only possible if none of the new features in the new version are used, for example a new key type that doesn't exist in the old version.

It seems pretty strait-forward to generate RDB of an older version. How to select the RDB version to output? There are some possibilities:

1. A new config rdb-version, default to "latest". The admin needs to set it to the required version before connecting the downgraded nodes.
2. Autodetect which RDB version to use. Since Valkey 8.0, replicas send REPLCONF version. If they don't, we can assume they're using a very old version, so we can pick let's say RDB 9, and we can use this for the full sync. There are some corner cases here though:
   • What if a new replica is already performing a full sync using the latest RDB and a downgraded replica connects? It can't share the already created RDB dump. Shall we create a new fork to do a new RDB dump with RDB 9, or shall we return some "Try again later" error?
   • How to handle RDB dump on disk? Which version should they use?

Option 1 (using a config) seems like the easiest and most explicit way.

[soloestoy]

Some of my concerns:

1. From the perspective of "official support," honestly, I don't really recommend supporting this feature. The official stance represents authority and standards, and I'm worried that once the downgrade feature is incorporated into the standard, it might be misused. Although our original intention is to allow rollbacks during upgrade issues, users could potentially use it in other scenarios, which is a common occurrence.

2. The development and maintenance cost of supporting downgrades is considerable. Based on the discussion above, if we are to be backward compatible with old RDB formats, it means retaining the code for generating old formats. A few versions might be manageable, but in the long run, the burden could be very heavy.

3. We haven't considered the aspect of incremental synchronization. Even if users aren't using new data structures or commands from the new version, downgrades might still fail. For example, in version 7.0, to improve expiration precision, set key value ex ttl is rewritten as set key value pxat timestamp, but some versions like 6.0 do not support the pxat parameter, which could cause downgrades to fail.

[zuiderkwast]

I understand it means extra maintenance burden.

We can discuss how many old versions we want to support. Maybe supporting one previous major version can be enough. To be able to undo an upgrade is a great improvement for critical software.

Regarding generating old RDB formats, it does not need to be identical to how it was done in the past. For example, we don't need to generate ziplist encoding. For sets, we currently use dictIterator to output sets with OBJ_ENCODING_HT, but if we use setTypeIterator instead, thi code it can output any encoding as RDB_TYPE_SET, including the listpack encoded sets, when we want to generate old RDB formats. Then, the code for generating old RDB format will just be a few if conditions.

It's a good finding that also command replication has some special cases. We definitely need cross-version testing...

[madolson]

@valkey-io/core-team Vote about this high level decision.

Support all minors of immediate previous major version. For example, if you are on version 9.0, you can can sync a replica from 8.1 and 8.2, but not 7.2. If the user is using any new features, the replication may get aborted by the replica. Lowest version supported will be 7.2.

[madolson]

Adding this to the 9.0, we will need to make a decision because we will bump the RDB version for Hash field expiration.

[jdork0]

Just adding another vote for this feature to be included. Our telco customer are always insisting on rollback capabilities in our deployments.

[hwware]

Just adding another vote for this feature to be included. Our telco customer are always insisting on rollback capabilities in our deployments.

You mean this vote? #1108 (comment)