Listpack encoding for sets

[zuiderkwast]

Small sets with not only integer elements are listpack encoded, by default up to 128 elements, max 64 bytes per element, new config set-max-listpack-entries and set-max-listpack-value. This saves memory for small sets compared to using a hashtable.

Sets with only integers, even very small sets, are still intset encoded (up to 1G limit, etc.). Larger sets are hashtable encoded.

Possible conversions when elements are added:

intset -> listpack
listpack -> hashtable
intset -> hashtable

Note: No conversion happens when elements are deleted. If all elements are deleted and then added again, the set is deleted and recreated, thus implicitly converted to a smaller encoding.

This PR increments the RDB version, and has an effect on OBJECT ENCODING

Fixes #11155.

Benchmark

redis-server started with default config on my laptop.

redis-benchmark -P 10 -q --threads 2 -n 10000000 -r 200000 COMMAND ARGS... was
used for benchmarking, where command and args are given as in the table below.

Throughput (requests per second)

Command	Unstable	This PR	Difference
sadd __rand_int__ __rand_int__	579407.81	655479.81	+13%
smembers __rand_int__	83266.02	220735.92	+165%
sismember __rand_int__ __rand_int__	676956.38	570580.88	-16%
spop __rand_int__	412014.34	563031.38	+37%

Memory usage

Memory usage after the 10M SADD commands, resulting in 200k keys with ~50 set
elements per key, checked using ./redis-cli info memory | grep ^used_memory_human shows memory usage down by 70%.

Command	Unstable	This PR	Difference
Memory usage 200k sets with ~50 elements	539.98M	160.19M	-70%

Remarks

The keys and the set elements given as __rand_int__ which results in strings
on the form "000000129245". Since they have leading zeroes, these strings are
not integer-encoded in a listpack. With numeric values without leading zeroes,
you can expect even greater memory savings. Huge memory saving is expected,
since listpack is a compact representation designed to save memory.

The huge speed improvement for SMEMBERS is expected, since a listpack is a
single allocation and traversing it means very few cache misses. Traversing the
keys in a hash table OTOH means several memory lookups.

The degradation for SISMEMBER can be expected, since a hashtable-lookup is
naturally faster than a linear search through a listpack of ~50 elements.

[sundb]

missing set-listpack in

redis/src/redis-check-rdb.c

Line 100 in 1de675b

"quicklist-v2"

[zuiderkwast]

missing set-listpack in redis-check-rdb.c

Done.

[oranagra]

@zuiderkwast thank you for taking the time.
here's my initial review (didn't look at the tests yet).

there are a bunch of minor comments here, but also some critical ones about the efficiency (specifically of random picks in listpack, and about popping them out).

p.s. eventually, this PR top comment should include some benchmark that mentions both memory savings and also throughput / latency costs.