On-demand database allocation instead of preallocation

[xbasel]

Allocate database structures lazily to prevent excessive memory usage
when a large number of databases is configured but not actually used.

fixes #1597

[ranshid]

some immediate comments.

[codecov]

Codecov Report

Attention: Patch coverage is 99.21875% with 1 line in your changes missing coverage. Please review.

Project coverage is 71.53%. Comparing base (3789b29) to head (57573e5).
Report is 10 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/replication.c	83.33%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1609      +/-   ##
============================================
+ Coverage     71.43%   71.53%   +0.09%     
============================================
  Files           122      122              
  Lines         66210    66484     +274     
============================================
+ Hits          47300    47561     +261     
- Misses        18910    18923      +13     

Files with missing lines	Coverage Δ
src/aof.c	80.40% <ø> (+0.08%)	⬆️
src/cluster.c	90.38% <100.00%> (ø)
src/cluster_legacy.c	86.79% <100.00%> (+0.07%)	⬆️
src/db.c	90.07% <100.00%> (-0.01%)	⬇️
src/debug.c	53.58% <100.00%> (+0.12%)	⬆️
src/defrag.c	81.59% <100.00%> (-1.04%)	⬇️
src/evict.c	98.50% <100.00%> (+0.02%)	⬆️
src/expire.c	96.62% <100.00%> (+0.02%)	⬆️
src/object.c	81.41% <100.00%> (ø)
src/rdb.c	76.89% <100.00%> (+0.56%)	⬆️
... and 3 more

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xbasel]

Running valkey with 10 million databases:

valkey-server --databases 10000000

empty dbs:

Before:

# Memory
used_memory:6086209848
used_memory_human:5.67G
used_memory_rss:6063915008
used_memory_rss_human:5.65G

After:

# Memory
used_memory:84790224
used_memory_human:80.86M
used_memory_rss:7602176
used_memory_rss_human:7.25M

db 8 million:

# Keyspace
db0:keys=1,expires=0,avg_ttl=0
db8000000:keys=1,expires=0,avg_ttl=0
127.0.0.1:6379[8000000]>

[zuiderkwast]

Great, this looks pretty strait-forward.

The benchmark with 10M databases is great, but not a very common I think. :) The main motivation for this optimization IMO is in combination with #1319. For cluster mode with 16 empty databases, does it go from 10MB to 1MB?

[xbasel]

Great, this looks pretty strait-forward.

The benchmark with 10M databases is great, but not a very common I think. :) The main motivation for this optimization IMO is in combination with #1319. For cluster mode with 16 empty databases, does it go from 10MB to 1MB?

This has been created artificially (this change hasn't been merged into multi-database code, i changed the code to create 16k slots without cluster mode)

before (preallocated) with (16 databases, 16k slots):

# Memory
used_memory:15887960
used_memory_human:15.15M
used_memory_rss:15204352
used_memory_rss_human:14.50M
used_memory_peak:15887960
used_memory_peak_human:15.15M

allocated-on-demand (16 databases, 16k slots), database 0 is always pre-allocated..

# Memory
used_memory:7327440
used_memory_human:6.99M
used_memory_rss:14548992
used_memory_rss_human:13.88M

[xbasel]

@zuiderkwast / @ranshid reminder :)

[zuiderkwast]

It's not forgotten, but the focus now is on stabilizing the 8.1 release and fix the flaky tests before 8.1 GA. This one will have to wait until after 8.1. It will be good to release it together with the related cluster-multidb feature.

[soloestoy]

I see many changes from server.db->keys to server.db[0]->keys, such as in countKeysInSlot. This will conflict with multi-databases. I think this PR should be aligned after the multi-databases merge.

[zuiderkwast]

Yeah this looks good to merge.