Skip to content

[Bug Fix] Optimize RDB Load Performance and Fix Cluster Mode Resizing #1199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 18, 2024
Merged

[Bug Fix] Optimize RDB Load Performance and Fix Cluster Mode Resizing #1199

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3d51cdd
On RDB load, after reading db size opcode forcefully expand main dict
naglera Oct 21, 2024
4723a4f
Update src/rdb.c
naglera Oct 22, 2024
465ded6
dict expand if missing by default
naglera Oct 22, 2024
a394c0a
Fix documentation
naglera Oct 28, 2024
3f70b22
Enable slot size estimation for legacy RDB load. The slot count shoul…
naglera Nov 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion src/kvstore.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,7 +424,8 @@ unsigned long long kvstoreScan(kvstore *kvs,
*/
int kvstoreExpand(kvstore *kvs, uint64_t newsize, int try_expand, kvstoreExpandShouldSkipDictIndex *skip_cb) {
for (int i = 0; i < kvs->num_dicts; i++) {
dict *d = kvstoreGetDict(kvs, i);
/* If the dictionary doesn't exist, create it */
dict *d = createDictIfNeeded(kvs, i);
if (!d || (skip_cb && skip_cb(i))) continue;
int result = try_expand ? dictTryExpand(d, newsize) : dictExpand(d, newsize);
if (try_expand && result == DICT_ERR) return 0;
Expand Down
11 changes: 8 additions & 3 deletions src/rdb.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3075,7 +3075,12 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin
* selected data base, in order to avoid useless rehashing. */
if ((db_size = rdbLoadLen(rdb, NULL)) == RDB_LENERR) goto eoferr;
if ((expires_size = rdbLoadLen(rdb, NULL)) == RDB_LENERR) goto eoferr;
should_expand_db = 1;
if (!server.cluster_enabled) {
/* Ignore RDB_OPCODE_RESIZEDB in Valkey cluster: Since Valkey 8, it uses dictionary per slot. Using resize opcode might
* expand all the replica's dictionaries, including dictionaries of unowned slots. This causes memory
* overhead when syncing from Valkey 7 primaries. */
should_expand_db = 1;
}
continue; /* Read next opcode. */
} else if (type == RDB_OPCODE_AUX) {
/* AUX: generic string-string fields. Use to add state to RDB
Expand Down Expand Up @@ -3245,8 +3250,8 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin
continue;
}

/* If there is no slot info, it means that it's either not cluster mode or we are trying to load legacy RDB
* file. In this case we want to estimate number of keys per slot and resize accordingly. */
/* If there is no slot info, it means that it's either not cluster mode or we are trying to load a legacy RDB
* file. In either cases we want to resize the db accordingly. */
if (should_expand_db) {
dbExpand(db, db_size, 0);
dbExpandExpires(db, expires_size, 0);
Expand Down
Loading