Skip to content

active expiry #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 10 commits into from
Jun 25, 2025
Merged

active expiry #2

merged 10 commits into from
Jun 25, 2025

Conversation

@xbasel
Copy link
Collaborator

@xbasel xbasel commented May 21, 2025

No description provided.

@xbasel xbasel force-pushed the ttl-poc-new branch 3 times, most recently from 33432b0 to 63fefe0 Compare June 17, 2025 05:55
ranshid added a commit that referenced this pull request Jun 18, 2025
@ranshid
Only mark the client reprocessing flag when unblocked on keys (#2… (v…
5dc6632 
…alkey-io#2231)

NOTE - this is a backport of
valkey-io#2109

When we refactored the blocking framework we introduced the client
reprocessing infrastructure. In cases the client was blocked on keys, it
will attempt to reprocess the command. One challenge was to keep track
of the command timeout, since we are reprocessing and do not want to
re-register the client with a fresh timeout each time. The solution was
to consider the client reprocessing flag when the client is
blockedOnKeys:

```
if (!c->flag.reprocessing_command) {
        /* If the client is re-processing the command, we do not set the timeout
         * because we need to retain the client's original timeout. */ c->bstate->timeout = timeout; } 
```

However, this introduced a new issue. There are cases where the client
will consecutive blocking of different types for example:
   
```
CLIENT PAUSE 10000 ALL
BZPOPMAX zset 1
```
would have the client blocked on the zset endlessly if nothing will be
written to it.

**Credits to @uriyage for locating this with his fuzzer testing**

The suggested solution is to only flag the client when it is
specifically unblocked on keys.

Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid ranshid force-pushed the ttl-poc-new branch 2 times, most recently from 65eeb1d to 8ecd584 Compare June 18, 2025 15:12
ranshid pushed a commit that referenced this pull request Jun 19, 2025
@Fusl @enjoy-binbin
Ensure empty error tables in scripts don't crash Valkey (valkey-io#2229)
04353ae 
When calling the command `EVAL error{} 0`, Valkey crashes with the
following stack trace. This patch ensures we never leave the
`err_info.msg` field null when we fail to extract a proper error
message.

```
=== VALKEY BUG REPORT START: Cut & paste starting from here ===
2595901:M 18 Jun 2025 01:20:12.917 # valkey 8.1.2 crashed by signal: 11, si_code: 1
2595901:M 18 Jun 2025 01:20:12.917 # Accessing address: (nil)
2595901:M 18 Jun 2025 01:20:12.917 # Crashed running the instruction at: 0x726f8e57ed1d

------ STACK TRACE ------
EIP:
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]

2595905 bio_aof
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595904 bio_close_file
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595901 valkey-server *
/usr/lib/libc.so.6(+0x3def0) [0x726f8e44def0]
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]
valkey-server *:6379(sdscatfmt+0x894) [0x6530abaa24a4]
valkey-server *:6379(luaCallFunction+0x39a) [0x6530abbc66ea]
valkey-server *:6379(+0x1a0992) [0x6530abbc6992]
valkey-server *:6379(scriptingEngineCallFunction+0x98) [0x6530abbc1298]
valkey-server *:6379(+0x11ff55) [0x6530abb45f55]
valkey-server *:6379(call+0x174) [0x6530aba94454]
valkey-server *:6379(processCommand+0x93d) [0x6530aba958dd]
valkey-server *:6379(processCommandAndResetClient+0x21) [0x6530abaa9d11]
valkey-server *:6379(processInputBuffer+0xe3) [0x6530abaaee83]
valkey-server *:6379(readQueryFromClient+0x65) [0x6530abaaef55]
valkey-server *:6379(+0x18e31a) [0x6530abbb431a]
valkey-server *:6379(aeProcessEvents+0x24a) [0x6530aba790ca]
valkey-server *:6379(aeMain+0x2d) [0x6530aba7938d]
valkey-server *:6379(main+0x3f6) [0x6530aba6e7b6]
/usr/lib/libc.so.6(+0x276b5) [0x726f8e4376b5]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x726f8e437769]
valkey-server *:6379(_start+0x25) [0x6530aba70235]

2595906 bio_lazy_free
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

4/4 expected stacktraces.

------ STACK TRACE DONE ------

------ REGISTERS ------
2595901:M 18 Jun 2025 01:20:12.920 # 
RAX:0000000000000000 RBX:0000726f8dd35663
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000010
RBP:00007ffc2b821a80 RSP:00007ffc2b821938
R8 :000000000000000c R9 :00006530abc111b8
R10:0000000000000001 R11:0000000000000003
R12:00006530abc49adc R13:00006530abc111b7
R14:0000000000000001 R15:0000000000000001
RIP:0000726f8e57ed1d EFL:0000000000010283
CSGSFS:002b000000000033
2595901:M 18 Jun 2025 01:20:12.921 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
server_name:valkey
valkey_version:8.1.2
valkey_release_stage:ga
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:38d65aa7b4148d2c
server_mode:standalone
os:Linux 6.14.6-arch1-1 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:15.1.1
process_id:2595901
process_supervised:no
run_id:a0b75f67a217a81142f17553028c010e86c1ee80
tcp_port:6379
server_time_usec:1750209612917634
uptime_in_seconds:16
uptime_in_days:0
hz:10
configured_hz:10
clients_hz:10
lru_clock:5379148
executable:/home/fusl/valkey-server
config_file:
io_threads_active:0
availability_zone:
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
paused_reason:none
paused_actions:none
paused_timeout_milliseconds:0

# Memory
used_memory:911824
used_memory_human:890.45K
used_memory_rss:15323136
used_memory_rss_human:14.61M
used_memory_peak:911824
used_memory_peak_human:890.45K
used_memory_peak_perc:100.29%
used_memory_overhead:892232
used_memory_startup:891824
used_memory_dataset:19592
used_memory_dataset_perc:97.96%
allocator_allocated:1845952
allocator_active:1986560
allocator_resident:6672384
allocator_muzzy:0
total_system_memory:67323842560
total_system_memory_human:62.70G
used_memory_lua:34816
used_memory_vm_eval:34816
used_memory_lua_human:34.00K
used_memory_scripts_eval:184
number_of_cached_scripts:1
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:68608
used_memory_vm_total_human:67.00K
used_memory_functions:224
used_memory_scripts:408
used_memory_scripts_human:408B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:0
allocator_rss_ratio:3.36
allocator_rss_bytes:4685824
rss_overhead_ratio:2.30
rss_overhead_bytes:8650752
mem_fragmentation_ratio:17.18
mem_fragmentation_bytes:14431168
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1750209596
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:34
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:1
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
io_threaded_freed_objects:0
io_threaded_accept_processed:0
io_threaded_poll_processed:0
io_threaded_total_prefetch_batches:0
io_threaded_total_prefetch_entries:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:170
eventloop_duration_sum:17739
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:9
instantaneous_eventloop_duration_usec:99
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:d35a0bb7979f490a60174bb363524431d7eb2428
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.012543
used_cpu_user:0.016853
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.012440
used_cpu_user_main_thread:0.016714

# Modules

# Commandstats

# Errorstats

# Latencystats

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0

------ CURRENT CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ EXECUTING CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
repl-diskless-load disabled
debug-context ""
sanitize-dump-payload no
lazyfree-lazy-user-del yes
lazyfree-lazy-server-del yes
import-mode no
lazyfree-lazy-user-flush yes
list-compress-depth 0
dual-channel-replication-enabled no
repl-diskless-sync yes
activedefrag no
lazyfree-lazy-expire yes
io-threads 1
replica-read-only yes
client-query-buffer-limit 1gb
slave-read-only yes
lazyfree-lazy-eviction yes
proto-max-bulk-len 512mb

------ FAST MEMORY TEST ------
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #0 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #1 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #2 terminated
*** Preparing to test memory region 6530abce2000 (212992 bytes)
*** Preparing to test memory region 726f8af7f000 (2621440 bytes)
*** Preparing to test memory region 726f8b200000 (8388608 bytes)
*** Preparing to test memory region 726f8ba00000 (4194304 bytes)
*** Preparing to test memory region 726f8bffe000 (8388608 bytes)
*** Preparing to test memory region 726f8c7ff000 (8388608 bytes)
*** Preparing to test memory region 726f8d000000 (8388608 bytes)
*** Preparing to test memory region 726f8dc00000 (4194304 bytes)
*** Preparing to test memory region 726f8e290000 (16384 bytes)
*** Preparing to test memory region 726f8e3d2000 (20480 bytes)
*** Preparing to test memory region 726f8e5f8000 (32768 bytes)
*** Preparing to test memory region 726f8eb58000 (12288 bytes)
*** Preparing to test memory region 726f8eb5c000 (16384 bytes)
*** Preparing to test memory region 726f8ed63000 (4096 bytes)
*** Preparing to test memory region 726f8eef2000 (397312 bytes)
*** Preparing to test memory region 726f8efc7000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/lib/libc.so.6 (base 0x726f8e410000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===
```

---------

Signed-off-by: Fusl <[email protected]>
Signed-off-by: Binbin <[email protected]>
Co-authored-by: Binbin <[email protected]>
@xbasel xbasel changed the title add basic test for notifications active expiry Jun 22, 2025
@xbasel xbasel force-pushed the ttl-poc-new branch 2 times, most recently from 15d67cb to 896798f Compare June 23, 2025 07:11
@ranshid ranshid closed this Jun 23, 2025
ranshid and others added 4 commits June 23, 2025 18:29
@ranshid
Introduce Hash fields active expiry
665eae3 
Squashed commit of the following:

commit f13c64e061876f441607fb02682b3b5edb8cce39
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 18:22:23 2025 +0300

    small fix

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 7ea26009b3cc8b04531ef7eb8904f81ec445061a
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 17:48:05 2025 +0300

    fix some more issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 0e3e4ce27eaeb8bff7919d39ba500ebab78d5486
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 15:41:37 2025 +0300

    ome more fixes

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 85ab2e424162e615ab642d9a7985143eb825edeb
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 15:33:47 2025 +0300

    fix tests after merge

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 11c7b05b34313d1723656c31e3d1de0e418dcc6a
Merge: bec831415 f8c7356
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 14:31:38 2025 +0300

    Merge remote-tracking branch 'valkey-fork/ttl-poc-new' into ttl-poc-new-xbasel

commit bec831415d071ab4a18fb604a1a9d88c06bbee1e
Merge: 1053af0d1 15d67cb
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 23 09:18:47 2025 +0300

    Merge remote-tracking branch 'xbasel/ttl-poc-new' into ttl-poc-new-xbasel

commit 1053af0d1cdc718a7899a5bcfb2a6cfb0a67a20b
Merge: 5d423faac cc59a7e
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 22 10:53:14 2025 +0300

    Merge remote-tracking branch 'valkey-fork/ttl-poc-new' into ttl-poc-new-xbasel

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 15d67cb
Author: xbasel <[email protected]>
Date:   Sun Jun 22 21:10:49 2025 +0300

    timed proc

    Signed-off-by: xbasel <[email protected]>

commit d811734
Author: xbasel <[email protected]>
Date:   Sun Jun 22 19:58:46 2025 +0300

    simplify iteration

    Signed-off-by: xbasel <[email protected]>

commit 96cd142
Author: xbasel <[email protected]>
Date:   Sun Jun 22 17:24:56 2025 +0300

    remove commented out code

    Signed-off-by: xbasel <[email protected]>

commit c00ab83
Author: xbasel <[email protected]>
Date:   Sun Jun 22 17:13:43 2025 +0300

    fix after rebase

    Signed-off-by: xbasel <[email protected]>

commit f30be2c
Author: xbasel <[email protected]>
Date:   Sun Jun 22 17:09:10 2025 +0300

    uncomment

    Signed-off-by: xbasel <[email protected]>

commit 72072be
Author: xbasel <[email protected]>
Date:   Sun Jun 22 17:08:08 2025 +0300

    add stats to info

    Signed-off-by: xbasel <[email protected]>

commit af1ad9a
Author: xbasel <[email protected]>
Date:   Sun Jun 22 13:14:22 2025 +0300

    hashTypeExpireEntry

    Signed-off-by: xbasel <[email protected]>

commit 785a1ff
Author: xbasel <[email protected]>
Date:   Sun Jun 22 12:35:14 2025 +0300

    volatile set iterator -> iterate over entires, not buckets

    Signed-off-by: xbasel <[email protected]>

commit 01e19bf
Author: xbasel <[email protected]>
Date:   Sun Jun 22 11:09:26 2025 +0300

    fixes

    Signed-off-by: xbasel <[email protected]>

commit 35db373
Author: xbasel <[email protected]>
Date:   Thu Jun 19 14:46:44 2025 +0300

    fix rebase issues

    Signed-off-by: xbasel <[email protected]>

commit 274f72f
Author: xbasel <[email protected]>
Date:   Wed Jun 18 20:58:08 2025 +0300

    m

    Signed-off-by: xbasel <[email protected]>

commit 85f53f7
Author: xbasel <[email protected]>
Date:   Tue Jun 17 18:23:54 2025 +0300

    mm

    Signed-off-by: xbasel <[email protected]>

commit daeb6f0
Author: xbasel <[email protected]>
Date:   Tue Jun 17 11:17:06 2025 +0300

    cleanup
    Signed-off-by: xbasel <[email protected]>

commit f7fac1b
Author: xbasel <[email protected]>
Date:   Tue Jun 17 10:22:40 2025 +0300

    more tests

    Signed-off-by: xbasel <[email protected]>

commit 5cc7714
Author: xbasel <[email protected]>
Date:   Sun Jun 8 20:05:33 2025 +0300

    hfe activeexpiry - draft

    Signed-off-by: xbasel <[email protected]>

commit 2f9a9e3
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 15 14:17:05 2025 +0300

    fix misspel in test

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 12e5c66
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 15 13:35:58 2025 +0300

    fix flakey test with EXPIREAT

    Signed-off-by: Ran Shidlansik <[email protected]>

commit f7fc8b1
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 12 17:17:58 2025 +0300

    add missing files

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 8f0d848
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 12 17:15:38 2025 +0300

    Separate hash entry implementation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit dfd0767
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 18:47:11 2025 +0300

    fix some bugs and added HPERSIST tests to help schema validator

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2569b42
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 9 22:02:39 2025 +0300

    remove fmacros include from volatile_set

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 86c099a
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 9 19:56:14 2025 +0300

    completely remove server level access context 9it was mainly used for lazy expiration logic)

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 50421c0
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 15:36:32 2025 +0300

    switch hashtable type only when object has volatile items

    Signed-off-by: Ran Shidlansik <[email protected]>

commit ec887e7
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 13:02:03 2025 +0300

    remove lazy expiration logic and tests

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 399533c
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 12:18:48 2025 +0300

     remove metadata from hash entry

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 6088a9f
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 19:41:59 2025 +0300

    make sure to remove the volatile set on hash object detructor

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 4c8bcc5
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 16:01:56 2025 +0300

    fix trackUpdate condition

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2fe863b
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 15:16:53 2025 +0300

    fix bad memory access issue on entry tracking update

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2c734e9
Author: xbasel <[email protected]>
Date:   Wed May 21 14:50:18 2025 +0300

    Hash TTL - add tests (valkey-io#1)

    * add tests

    Signed-off-by: xbasel <[email protected]>

    * fix a bug - return on error

    Signed-off-by: xbasel <[email protected]>

    * disable failing tests

    Signed-off-by: xbasel <[email protected]>

    * rmeove redundant test

    Signed-off-by: xbasel <[email protected]>

    * Update tests/unit/hashexpire.tcl

    ---------

    Signed-off-by: xbasel <[email protected]>
    Co-authored-by: Ran Shidlansik <[email protected]>

commit 62e515e
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 14:43:49 2025 +0300

    make hashtable call entry destructor on delete access

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c375237
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 14:31:44 2025 +0300

    centralize keyspace and key signal notifications to the reset context

    Signed-off-by: Ran Shidlansik <[email protected]>

commit d30380c
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:47:42 2025 +0300

    fix formatting issue

    Signed-off-by: Ran Shidlansik <[email protected]>

commit e002ca6
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:25:17 2025 +0300

    fix build issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 98e9a8a
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:13:15 2025 +0300

    allow setting the key object in context

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 94b345c
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 20:13:23 2025 +0300

    fix hexpire propagation to use hpexpireat

    Signed-off-by: Ran Shidlansik <[email protected]>

commit fa08915
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 17:19:41 2025 +0300

    fix HGETEX replication handling

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 022fcfc
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:46:31 2025 +0300

    make httl functions verify the type

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c2a48a4
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:24:59 2025 +0300

    Fix HEXPIRE parse limits

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 4759369
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:19:53 2025 +0300

    fix FNX/FXX logic

    Signed-off-by: Ran Shidlansik <[email protected]>

commit f1d7d38
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:08:02 2025 +0300

    fix wrong assert condition on update entry

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 8e0dc43
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:00:20 2025 +0300

    handle negative ttl correctly

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2b0a6b0
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 14:17:19 2025 +0300

    Add support for HGETEX and HSETEX

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c4e1c26
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 11:12:49 2025 +0300

    remove hashtable redundant log

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 00b18c8
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 10:34:27 2025 +0300

    avoid extra ref count incrementing in hashTypePropagateDeletion

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c8c9422
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 21:30:46 2025 +0300

    fix typo

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 6bde139
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 21:28:02 2025 +0300

    fix expire propagation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 9482444
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 20:38:21 2025 +0300

    fix new introduced commands

    Signed-off-by: Ran Shidlansik <[email protected]>

commit f517983
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jan 6 10:46:47 2025 +0200

    Introduce HASH items expiration

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 5d423faace3a558e2fda580559d8ffbe4e3ee6d7
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 17 21:19:31 2025 +0300

    WIP

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 0b4f160
Author: xbasel <[email protected]>
Date:   Tue Jun 17 11:17:06 2025 +0300

    cleanup
    Signed-off-by: xbasel <[email protected]>

commit 4d0fd94
Author: xbasel <[email protected]>
Date:   Tue Jun 17 10:22:40 2025 +0300

    more tests

    Signed-off-by: xbasel <[email protected]>

commit 63fefe0
Author: xbasel <[email protected]>
Date:   Sun Jun 8 20:05:33 2025 +0300

    hfe activeexpiry - draft

    Signed-off-by: xbasel <[email protected]>

commit cd674be
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 15 14:17:05 2025 +0300

    fix misspel in test

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 25954b3
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 15 13:35:58 2025 +0300

    fix flakey test with EXPIREAT

    Signed-off-by: Ran Shidlansik <[email protected]>

commit aee670e
Author: Ran Shidlansik <[email protected]>
Date:   Sun Jun 15 11:36:27 2025 +0300

    fix some more memory leaks

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2fea8e7
Author: Ran Shidlansik <[email protected]>
Date:   Fri Jun 13 15:02:13 2025 +0300

    fix memory leak issue

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 56db999
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 12 17:21:51 2025 +0300

    fix bad compilation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 1b1ce58
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 12 17:17:58 2025 +0300

    add missing files

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 7a89a70
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 12 17:15:38 2025 +0300

    Separate hash entry implementation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 23bb4a2
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 19:25:32 2025 +0300

    extend the comment

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2a5e9a2
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 18:52:53 2025 +0300

    fix merge issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit e7683b6
Merge: 12151e5 c41ffc3
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 18:52:18 2025 +0300

    Merge remote-tracking branch 'origin/unstable' into ttl-poc-new

commit 12151e5
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 18:47:11 2025 +0300

    fix some bugs and added HPERSIST tests to help schema validator

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 846f943
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 14:06:58 2025 +0300

    fix new commands json and enable silent tests

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 51f9bdc
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 11:40:09 2025 +0300

    better enforce fields number to match the number of provided fields in httl and hpersist commands

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c1cefec
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 10:17:31 2025 +0300

    fix reply schema of commands fetching the hash field ttl

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 4db5b7c
Author: Ran Shidlansik <[email protected]>
Date:   Tue Jun 10 09:47:05 2025 +0300

    fix hexpire flaky test

    Signed-off-by: Ran Shidlansik <[email protected]>

commit f4ae1a2
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 9 22:02:39 2025 +0300

    remove fmacros include from volatile_set

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 3190eb4
Merge: 99d25d3 1941d28
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 9 21:15:07 2025 +0300

    Merge remote-tracking branch 'origin/unstable' into ttl-poc-new

commit 99d25d3
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jun 9 19:56:14 2025 +0300

    completely remove server level access context 9it was mainly used for lazy expiration logic)

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2e082db
Author: Ran Shidlansik <[email protected]>
Date:   Thu Jun 5 09:34:16 2025 +0300

    exlude hexpire tests from external tests

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 2c4c312
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 15:36:32 2025 +0300

    switch hashtable type only when object has volatile items

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 124acbe
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 14:45:19 2025 +0300

    return syntax error when fields is not provided in new API arguments

    Signed-off-by: Ran Shidlansik <[email protected]>

commit dd071f9
Merge: 12a35e2 5699c8c
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 13:02:30 2025 +0300

    Merge remote-tracking branch 'origin/unstable' into ttl-poc-new

commit 12a35e2
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 13:02:03 2025 +0300

    remove lazy expiration logic and tests

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 0cadaec
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 12:27:39 2025 +0300

    copy hash object should also copy the fields ttl

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 79b7e78
Author: Ran Shidlansik <[email protected]>
Date:   Wed Jun 4 12:18:48 2025 +0300

     remove metadata from hash entry

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 8654080
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 19:41:59 2025 +0300

    make sure to remove the volatile set on hash object detructor

    Signed-off-by: Ran Shidlansik <[email protected]>

commit eab6fc4
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 16:01:56 2025 +0300

    fix trackUpdate condition

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 6d9551e
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 15:16:53 2025 +0300

    fix bad memory access issue on entry tracking update

    Signed-off-by: Ran Shidlansik <[email protected]>

commit d719dcd
Author: xbasel <[email protected]>
Date:   Wed May 21 14:50:18 2025 +0300

    Hash TTL - add tests (valkey-io#1)

    * add tests

    Signed-off-by: xbasel <[email protected]>

    * fix a bug - return on error

    Signed-off-by: xbasel <[email protected]>

    * disable failing tests

    Signed-off-by: xbasel <[email protected]>

    * rmeove redundant test

    Signed-off-by: xbasel <[email protected]>

    * Update tests/unit/hashexpire.tcl

    ---------

    Signed-off-by: xbasel <[email protected]>
    Co-authored-by: Ran Shidlansik <[email protected]>

commit e604b37
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 14:43:49 2025 +0300

    make hashtable call entry destructor on delete access

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 0b8dc03
Author: Ran Shidlansik <[email protected]>
Date:   Wed May 21 14:31:44 2025 +0300

    centralize keyspace and key signal notifications to the reset context

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 753ba3c
Author: Ran Shidlansik <[email protected]>
Date:   Tue May 20 14:19:34 2025 +0300

    fix object pass to keyspace notification in HSETEX

    Signed-off-by: Ran Shidlansik <[email protected]>

commit c5b8d76
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:47:42 2025 +0300

    fix formatting issue

    Signed-off-by: Ran Shidlansik <[email protected]>

commit e72d7a6
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:25:17 2025 +0300

    fix build issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 36b7356
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 21:13:15 2025 +0300

    allow setting the key object in context

    Signed-off-by: Ran Shidlansik <[email protected]>

commit a6844ac
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 20:15:18 2025 +0300

    add commands json files

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 20c0d29
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 20:13:23 2025 +0300

    fix hexpire propagation to use hpexpireat

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 5e19c90
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 17:19:41 2025 +0300

    fix HGETEX replication handling

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 31923c5
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:46:31 2025 +0300

    make httl functions verify the type

    Signed-off-by: Ran Shidlansik <[email protected]>

commit b782d44
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:36:22 2025 +0300

    fix case of hll command issues on non-existing listpack encoded hash

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 0723625
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:24:59 2025 +0300

    Fix HEXPIRE parse limits

    Signed-off-by: Ran Shidlansik <[email protected]>

commit d97e23f
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:19:53 2025 +0300

    fix FNX/FXX logic

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 4301399
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:08:02 2025 +0300

    fix wrong assert condition on update entry

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 6465314
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 15:00:20 2025 +0300

    handle negative ttl correctly

    Signed-off-by: Ran Shidlansik <[email protected]>

commit f62c163
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 14:20:49 2025 +0300

    format fixes

    Signed-off-by: Ran Shidlansik <[email protected]>

commit a59f31a
Author: Ran Shidlansik <[email protected]>
Date:   Mon May 19 14:17:19 2025 +0300

    Add support for HGETEX and HSETEX

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 4a09f3d
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 12:16:44 2025 +0300

    free entry when calling hashTypeDelete

    Signed-off-by: Ran Shidlansik <[email protected]>

commit dd62037
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 11:12:49 2025 +0300

    remove hashtable redundant log

    Signed-off-by: Ran Shidlansik <[email protected]>

commit ea039c4
Merge: fce9a43 8d686dd
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 10:40:39 2025 +0300

    Merge remote-tracking branch 'origin/unstable' into ttl-poc-new

commit fce9a43
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 10:39:22 2025 +0300

    fix cmake compilation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 1f0c933
Author: Ran Shidlansik <[email protected]>
Date:   Sun May 18 10:34:27 2025 +0300

    avoid extra ref count incrementing in hashTypePropagateDeletion

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 6ee497c
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 21:32:30 2025 +0300

    fix some more format issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 90b7536
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 21:30:46 2025 +0300

    fix typo

    Signed-off-by: Ran Shidlansik <[email protected]>

commit fcce92b
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 21:28:02 2025 +0300

    fix expire propagation

    Signed-off-by: Ran Shidlansik <[email protected]>

commit cc7c2a3
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 20:48:12 2025 +0300

    handle some format check issues

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 61bd39a
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 20:45:18 2025 +0300

    fix some spelling checks

    Signed-off-by: Ran Shidlansik <[email protected]>

commit 89f56b0
Author: Ran Shidlansik <[email protected]>
Date:   Thu May 15 20:38:21 2025 +0300

    fix new introduced commands

    Signed-off-by: Ran Shidlansik <[email protected]>

commit ecdcce0
Author: Ran Shidlansik <[email protected]>
Date:   Mon Jan 6 10:46:47 2025 +0200

    Introduce HASH items expiration

    Signed-off-by: Ran Shidlansik <[email protected]>

Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid
fix some merge dead code
93ab688 
Signed-off-by: Ran Shidlansik <[email protected]>
@xbasel
resotre src/Makefile
f6f18e8 
Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid
fix some bugs in volatile_set and write some unittests
539fe4e 
Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid ranshid reopened this Jun 24, 2025
ranshid and others added 6 commits June 24, 2025 08:57
@ranshid
Merge remote-tracking branch 'xbasel/ttl-poc-new' into introduce-fiel…
d426878 
…ds-active-axpiry
@xbasel
active expiry proc
b65a1a8 
Signed-off-by: xbasel <[email protected]>
@xbasel
Abort active expiry if all dbs were processed
56d1041 
Signed-off-by: xbasel <[email protected]>
@ranshid
tag buckets
e900e52 
Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid
Merge remote-tracking branch 'xbasel/ttl-poc-new' into introduce-fiel…
156be8c 
…ds-active-axpiry
@ranshid
1. fix bug in volatile_vector - correctly identify when we need to cr…
18c112a 
…eate

   new bucket when inserting entry
2. introduce volatileSetExpiredFirst and volatileSetExpiredPop

Signed-off-by: Ran Shidlansik <[email protected]>
@ranshid ranshid merged commit 18c112a into ranshid:ttl-poc-new Jun 25, 2025
6 of 18 checks passed
ranshid pushed a commit that referenced this pull request Jul 1, 2025
@gusakovy
Fix hashtablePauseAutoShrink and use it in hashtableScanDefrag (valke…
ad2c6b0 
…y-io#2257)

**Current state**
During `hashtableScanDefrag`, rehashing is paused to prevent entries
from moving, but the scan callback can still delete entries which
triggers `hashtableShrinkIfNeeded`. For example, the
`expireScanCallback` can delete expired entries.

**Issue**
This can cause the table to be resized and the old memory to be freed
while the scan is still accessing it, resulting in the following memory
access violation:

```
[err]: Sanitizer error: =================================================================
==46774==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000003100 at pc 0x0000004704d3 bp 0x7fffcb062000 sp 0x7fffcb061ff0
READ of size 1 at 0x611000003100 thread T0
    #0 0x4704d2 in isPositionFilled /home/gusakovy/Projects/valkey/src/hashtable.c:422
    #1 0x478b45 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1768
    #2 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    #3 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    #4 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    #5 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    #6 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    #7 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#8 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#9 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#10 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#11 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)
    valkey-io#12 0x452e39 in _start (/local/home/gusakovy/Projects/valkey/src/valkey-server+0x452e39)

0x611000003100 is located 0 bytes inside of 256-byte region [0x611000003100,0x611000003200)
freed by thread T0 here:
    #0 0x7f471a34a1e5 in __interceptor_free (/lib64/libasan.so.4+0xd81e5)
    #1 0x4aefbc in zfree_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:400
    #2 0x4aeff5 in valkey_free /home/gusakovy/Projects/valkey/src/zmalloc.c:415
    #3 0x4707d2 in rehashingCompleted /home/gusakovy/Projects/valkey/src/hashtable.c:456
    #4 0x471b5b in resize /home/gusakovy/Projects/valkey/src/hashtable.c:656
    #5 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #6 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #7 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    valkey-io#8 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    valkey-io#9 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    valkey-io#10 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    valkey-io#11 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    valkey-io#12 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    valkey-io#13 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    valkey-io#14 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    valkey-io#15 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    valkey-io#16 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    valkey-io#17 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    valkey-io#18 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    valkey-io#19 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    valkey-io#20 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#21 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#22 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#23 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#24 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

previously allocated by thread T0 here:
    #0 0x7f471a34a753 in __interceptor_calloc (/lib64/libasan.so.4+0xd8753)
    #1 0x4ae48c in ztrycalloc_usable_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:214
    #2 0x4ae757 in valkey_calloc /home/gusakovy/Projects/valkey/src/zmalloc.c:257
    #3 0x4718fc in resize /home/gusakovy/Projects/valkey/src/hashtable.c:645
    #4 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #5 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #6 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    #7 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    valkey-io#8 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    valkey-io#9 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    valkey-io#10 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    valkey-io#11 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    valkey-io#12 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    valkey-io#13 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    valkey-io#14 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    valkey-io#15 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    valkey-io#16 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    valkey-io#17 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    valkey-io#18 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    valkey-io#19 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#20 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#21 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#22 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#23 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

SUMMARY: AddressSanitizer: heap-use-after-free /home/gusakovy/Projects/valkey/src/hashtable.c:422 in isPositionFilled
Shadow bytes around the buggy address:
  0x0c227fff85d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85f0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8610: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
=>0x0c227fff8620:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8630: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8640: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==46774==ABORTING
```


**Solution**
Suggested solution is to also pause auto shrinking during
`hashtableScanDefrag`. I noticed that there was already a
`hashtablePauseAutoShrink` method and `pause_auto_shrink` counter, but
it wasn't actually used in `hashtableShrinkIfNeeded` so I fixed that.

**Testing**
I created a simple tcl test that (most of the times) triggers this
error, but it's a little clunky so I didn't add it as part of the PR:

```
start_server {tags {"expire hashtable defrag"}} {
    test {hashtable scan defrag on expiry} {

        r config set hz 100

        set num_keys 20
        for {set i 0} {$i < $num_keys} {incr i} {
            r set "key_$i" "value_$i"
        }

        for {set j 0} {$j < 50} {incr j} {
            set expire_keys 100
            for {set i 0} {$i < $expire_keys} {incr i} {
                # Short expiry time to ensure they expire quickly
                r psetex "expire_key_${i}_${j}" 100 "expire_value_${i}_${j}"
            }

            # Verify keys are set
            set initial_size [r dbsize]
            assert_equal $initial_size [expr $num_keys + $expire_keys]
            
            after 150
            for {set i 0} {$i < 10} {incr i} {
                r get "expire_key_${i}_${j}"
                after 10
            }
        }

        set remaining_keys [r dbsize]
        assert_equal $remaining_keys $num_keys

        # Verify server is still responsive
        assert_equal [r ping] {PONG}
    } {}
}
```
Compiling with ASAN using `make noopt SANITIZER=address valkey-server`
and running the test causes error above. Applying the fix resolves the
issue.

Signed-off-by: Yakov Gusakov <[email protected]>
ranshid pushed a commit that referenced this pull request Sep 30, 2025
@Fusl @enjoy-binbin @ranshid
Ensure empty error tables in scripts don't crash Valkey (valkey-io#2229)
551e209 
When calling the command `EVAL error{} 0`, Valkey crashes with the
following stack trace. This patch ensures we never leave the
`err_info.msg` field null when we fail to extract a proper error
message.

```
=== VALKEY BUG REPORT START: Cut & paste starting from here ===
2595901:M 18 Jun 2025 01:20:12.917 # valkey 8.1.2 crashed by signal: 11, si_code: 1
2595901:M 18 Jun 2025 01:20:12.917 # Accessing address: (nil)
2595901:M 18 Jun 2025 01:20:12.917 # Crashed running the instruction at: 0x726f8e57ed1d

------ STACK TRACE ------
EIP:
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]

2595905 bio_aof
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595904 bio_close_file
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

2595901 valkey-server *
/usr/lib/libc.so.6(+0x3def0) [0x726f8e44def0]
/usr/lib/libc.so.6(+0x16ed1d) [0x726f8e57ed1d]
valkey-server *:6379(sdscatfmt+0x894) [0x6530abaa24a4]
valkey-server *:6379(luaCallFunction+0x39a) [0x6530abbc66ea]
valkey-server *:6379(+0x1a0992) [0x6530abbc6992]
valkey-server *:6379(scriptingEngineCallFunction+0x98) [0x6530abbc1298]
valkey-server *:6379(+0x11ff55) [0x6530abb45f55]
valkey-server *:6379(call+0x174) [0x6530aba94454]
valkey-server *:6379(processCommand+0x93d) [0x6530aba958dd]
valkey-server *:6379(processCommandAndResetClient+0x21) [0x6530abaa9d11]
valkey-server *:6379(processInputBuffer+0xe3) [0x6530abaaee83]
valkey-server *:6379(readQueryFromClient+0x65) [0x6530abaaef55]
valkey-server *:6379(+0x18e31a) [0x6530abbb431a]
valkey-server *:6379(aeProcessEvents+0x24a) [0x6530aba790ca]
valkey-server *:6379(aeMain+0x2d) [0x6530aba7938d]
valkey-server *:6379(main+0x3f6) [0x6530aba6e7b6]
/usr/lib/libc.so.6(+0x276b5) [0x726f8e4376b5]
/usr/lib/libc.so.6(__libc_start_main+0x89) [0x726f8e437769]
valkey-server *:6379(_start+0x25) [0x6530aba70235]

2595906 bio_lazy_free
/usr/lib/libc.so.6(+0x9de22) [0x726f8e4ade22]
/usr/lib/libc.so.6(+0x91fda) [0x726f8e4a1fda]
/usr/lib/libc.so.6(+0x9264c) [0x726f8e4a264c]
/usr/lib/libc.so.6(pthread_cond_wait+0x14e) [0x726f8e4a4d1e]
valkey-server *:6379(bioProcessBackgroundJobs+0x1b4) [0x6530abb46db4]
/usr/lib/libc.so.6(+0x957eb) [0x726f8e4a57eb]
/usr/lib/libc.so.6(+0x11918c) [0x726f8e52918c]

4/4 expected stacktraces.

------ STACK TRACE DONE ------

------ REGISTERS ------
2595901:M 18 Jun 2025 01:20:12.920 # 
RAX:0000000000000000 RBX:0000726f8dd35663
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000010
RBP:00007ffc2b821a80 RSP:00007ffc2b821938
R8 :000000000000000c R9 :00006530abc111b8
R10:0000000000000001 R11:0000000000000003
R12:00006530abc49adc R13:00006530abc111b7
R14:0000000000000001 R15:0000000000000001
RIP:0000726f8e57ed1d EFL:0000000000010283
CSGSFS:002b000000000033
2595901:M 18 Jun 2025 01:20:12.921 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
server_name:valkey
valkey_version:8.1.2
valkey_release_stage:ga
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:38d65aa7b4148d2c
server_mode:standalone
os:Linux 6.14.6-arch1-1 x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:15.1.1
process_id:2595901
process_supervised:no
run_id:a0b75f67a217a81142f17553028c010e86c1ee80
tcp_port:6379
server_time_usec:1750209612917634
uptime_in_seconds:16
uptime_in_days:0
hz:10
configured_hz:10
clients_hz:10
lru_clock:5379148
executable:/home/fusl/valkey-server
config_file:
io_threads_active:0
availability_zone:
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
pubsub_clients:0
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
paused_reason:none
paused_actions:none
paused_timeout_milliseconds:0

# Memory
used_memory:911824
used_memory_human:890.45K
used_memory_rss:15323136
used_memory_rss_human:14.61M
used_memory_peak:911824
used_memory_peak_human:890.45K
used_memory_peak_perc:100.29%
used_memory_overhead:892232
used_memory_startup:891824
used_memory_dataset:19592
used_memory_dataset_perc:97.96%
allocator_allocated:1845952
allocator_active:1986560
allocator_resident:6672384
allocator_muzzy:0
total_system_memory:67323842560
total_system_memory_human:62.70G
used_memory_lua:34816
used_memory_vm_eval:34816
used_memory_lua_human:34.00K
used_memory_scripts_eval:184
number_of_cached_scripts:1
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:68608
used_memory_vm_total_human:67.00K
used_memory_functions:224
used_memory_scripts:408
used_memory_scripts_human:408B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:0
allocator_rss_ratio:3.36
allocator_rss_bytes:4685824
rss_overhead_ratio:2.30
rss_overhead_bytes:8650752
mem_fragmentation_ratio:17.18
mem_fragmentation_bytes:14431168
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:0
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1750209596
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:1
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:34
total_net_output_bytes:0
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:1
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
io_threaded_freed_objects:0
io_threaded_accept_processed:0
io_threaded_poll_processed:0
io_threaded_total_prefetch_batches:0
io_threaded_total_prefetch_entries:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:0
reply_buffer_expands:0
eventloop_cycles:170
eventloop_duration_sum:17739
eventloop_duration_cmd_sum:0
instantaneous_eventloop_cycles_per_sec:9
instantaneous_eventloop_duration_usec:99
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:0
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:d35a0bb7979f490a60174bb363524431d7eb2428
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:10485760
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.012543
used_cpu_user:0.016853
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.012440
used_cpu_user_main_thread:0.016714

# Modules

# Commandstats

# Errorstats

# Latencystats

# Cluster
cluster_enabled:0

# Keyspace

------ CLIENT LIST OUTPUT ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0

------ CURRENT CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ EXECUTING CLIENT INFO ------
id=2 addr=127.0.0.1:41372 laddr=127.0.0.1:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=12 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17060 events=r cmd=eval user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=34 tot-net-out=0 tot-cmds=0
argc: 3
argv[0]: "eval"
argv[1]: 7 bytes
argv[2]: 1 bytes

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
repl-diskless-load disabled
debug-context ""
sanitize-dump-payload no
lazyfree-lazy-user-del yes
lazyfree-lazy-server-del yes
import-mode no
lazyfree-lazy-user-flush yes
list-compress-depth 0
dual-channel-replication-enabled no
repl-diskless-sync yes
activedefrag no
lazyfree-lazy-expire yes
io-threads 1
replica-read-only yes
client-query-buffer-limit 1gb
slave-read-only yes
lazyfree-lazy-eviction yes
proto-max-bulk-len 512mb

------ FAST MEMORY TEST ------
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #0 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #1 terminated
2595901:M 18 Jun 2025 01:20:12.921 # Bio worker thread #2 terminated
*** Preparing to test memory region 6530abce2000 (212992 bytes)
*** Preparing to test memory region 726f8af7f000 (2621440 bytes)
*** Preparing to test memory region 726f8b200000 (8388608 bytes)
*** Preparing to test memory region 726f8ba00000 (4194304 bytes)
*** Preparing to test memory region 726f8bffe000 (8388608 bytes)
*** Preparing to test memory region 726f8c7ff000 (8388608 bytes)
*** Preparing to test memory region 726f8d000000 (8388608 bytes)
*** Preparing to test memory region 726f8dc00000 (4194304 bytes)
*** Preparing to test memory region 726f8e290000 (16384 bytes)
*** Preparing to test memory region 726f8e3d2000 (20480 bytes)
*** Preparing to test memory region 726f8e5f8000 (32768 bytes)
*** Preparing to test memory region 726f8eb58000 (12288 bytes)
*** Preparing to test memory region 726f8eb5c000 (16384 bytes)
*** Preparing to test memory region 726f8ed63000 (4096 bytes)
*** Preparing to test memory region 726f8eef2000 (397312 bytes)
*** Preparing to test memory region 726f8efc7000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /usr/lib/libc.so.6 (base 0x726f8e410000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===
```

---------

Signed-off-by: Fusl <[email protected]>
Signed-off-by: Binbin <[email protected]>
Co-authored-by: Binbin <[email protected]>
ranshid pushed a commit that referenced this pull request Sep 30, 2025
@gusakovy @ranshid
Fix hashtablePauseAutoShrink and use it in hashtableScanDefrag (valke…
4f12deb 
…y-io#2257)

**Current state**
During `hashtableScanDefrag`, rehashing is paused to prevent entries
from moving, but the scan callback can still delete entries which
triggers `hashtableShrinkIfNeeded`. For example, the
`expireScanCallback` can delete expired entries.

**Issue**
This can cause the table to be resized and the old memory to be freed
while the scan is still accessing it, resulting in the following memory
access violation:

```
[err]: Sanitizer error: =================================================================
==46774==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000003100 at pc 0x0000004704d3 bp 0x7fffcb062000 sp 0x7fffcb061ff0
READ of size 1 at 0x611000003100 thread T0
    #0 0x4704d2 in isPositionFilled /home/gusakovy/Projects/valkey/src/hashtable.c:422
    #1 0x478b45 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1768
    #2 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    #3 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    #4 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    #5 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    #6 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    #7 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#8 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#9 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#10 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#11 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)
    valkey-io#12 0x452e39 in _start (/local/home/gusakovy/Projects/valkey/src/valkey-server+0x452e39)

0x611000003100 is located 0 bytes inside of 256-byte region [0x611000003100,0x611000003200)
freed by thread T0 here:
    #0 0x7f471a34a1e5 in __interceptor_free (/lib64/libasan.so.4+0xd81e5)
    #1 0x4aefbc in zfree_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:400
    #2 0x4aeff5 in valkey_free /home/gusakovy/Projects/valkey/src/zmalloc.c:415
    #3 0x4707d2 in rehashingCompleted /home/gusakovy/Projects/valkey/src/hashtable.c:456
    #4 0x471b5b in resize /home/gusakovy/Projects/valkey/src/hashtable.c:656
    #5 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #6 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #7 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    valkey-io#8 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    valkey-io#9 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    valkey-io#10 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    valkey-io#11 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    valkey-io#12 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    valkey-io#13 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    valkey-io#14 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    valkey-io#15 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    valkey-io#16 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    valkey-io#17 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    valkey-io#18 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    valkey-io#19 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    valkey-io#20 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#21 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#22 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#23 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#24 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

previously allocated by thread T0 here:
    #0 0x7f471a34a753 in __interceptor_calloc (/lib64/libasan.so.4+0xd8753)
    #1 0x4ae48c in ztrycalloc_usable_internal /home/gusakovy/Projects/valkey/src/zmalloc.c:214
    #2 0x4ae757 in valkey_calloc /home/gusakovy/Projects/valkey/src/zmalloc.c:257
    #3 0x4718fc in resize /home/gusakovy/Projects/valkey/src/hashtable.c:645
    #4 0x475bff in hashtableShrinkIfNeeded /home/gusakovy/Projects/valkey/src/hashtable.c:1272
    #5 0x47704b in hashtablePop /home/gusakovy/Projects/valkey/src/hashtable.c:1448
    #6 0x47716f in hashtableDelete /home/gusakovy/Projects/valkey/src/hashtable.c:1459
    #7 0x480038 in kvstoreHashtableDelete /home/gusakovy/Projects/valkey/src/kvstore.c:847
    valkey-io#8 0x50c12c in dbGenericDeleteWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:490
    valkey-io#9 0x515f28 in deleteExpiredKeyAndPropagateWithDictIndex /home/gusakovy/Projects/valkey/src/db.c:1831
    valkey-io#10 0x516103 in deleteExpiredKeyAndPropagate /home/gusakovy/Projects/valkey/src/db.c:1844
    valkey-io#11 0x6d8642 in activeExpireCycleTryExpire /home/gusakovy/Projects/valkey/src/expire.c:70
    valkey-io#12 0x6d8706 in expireScanCallback /home/gusakovy/Projects/valkey/src/expire.c:139
    valkey-io#13 0x478bd8 in hashtableScanDefrag /home/gusakovy/Projects/valkey/src/hashtable.c:1770
    valkey-io#14 0x4789c2 in hashtableScan /home/gusakovy/Projects/valkey/src/hashtable.c:1729
    valkey-io#15 0x47e3ca in kvstoreScan /home/gusakovy/Projects/valkey/src/kvstore.c:402
    valkey-io#16 0x6d9040 in activeExpireCycle /home/gusakovy/Projects/valkey/src/expire.c:297
    valkey-io#17 0x4859d2 in databasesCron /home/gusakovy/Projects/valkey/src/server.c:1269
    valkey-io#18 0x486e92 in serverCron /home/gusakovy/Projects/valkey/src/server.c:1577
    valkey-io#19 0x4637dd in processTimeEvents /home/gusakovy/Projects/valkey/src/ae.c:370
    valkey-io#20 0x4643e3 in aeProcessEvents /home/gusakovy/Projects/valkey/src/ae.c:513
    valkey-io#21 0x4647ea in aeMain /home/gusakovy/Projects/valkey/src/ae.c:543
    valkey-io#22 0x4a61fc in main /home/gusakovy/Projects/valkey/src/server.c:7291
    valkey-io#23 0x7f471957c139 in __libc_start_main (/lib64/libc.so.6+0x21139)

SUMMARY: AddressSanitizer: heap-use-after-free /home/gusakovy/Projects/valkey/src/hashtable.c:422 in isPositionFilled
Shadow bytes around the buggy address:
  0x0c227fff85d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff85f0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c227fff8600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8610: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
=>0x0c227fff8620:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8630: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c227fff8640: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c227fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==46774==ABORTING
```


**Solution**
Suggested solution is to also pause auto shrinking during
`hashtableScanDefrag`. I noticed that there was already a
`hashtablePauseAutoShrink` method and `pause_auto_shrink` counter, but
it wasn't actually used in `hashtableShrinkIfNeeded` so I fixed that.

**Testing**
I created a simple tcl test that (most of the times) triggers this
error, but it's a little clunky so I didn't add it as part of the PR:

```
start_server {tags {"expire hashtable defrag"}} {
    test {hashtable scan defrag on expiry} {

        r config set hz 100

        set num_keys 20
        for {set i 0} {$i < $num_keys} {incr i} {
            r set "key_$i" "value_$i"
        }

        for {set j 0} {$j < 50} {incr j} {
            set expire_keys 100
            for {set i 0} {$i < $expire_keys} {incr i} {
                # Short expiry time to ensure they expire quickly
                r psetex "expire_key_${i}_${j}" 100 "expire_value_${i}_${j}"
            }

            # Verify keys are set
            set initial_size [r dbsize]
            assert_equal $initial_size [expr $num_keys + $expire_keys]
            
            after 150
            for {set i 0} {$i < 10} {incr i} {
                r get "expire_key_${i}_${j}"
                after 10
            }
        }

        set remaining_keys [r dbsize]
        assert_equal $remaining_keys $num_keys

        # Verify server is still responsive
        assert_equal [r ping] {PONG}
    } {}
}
```
Compiling with ASAN using `make noopt SANITIZER=address valkey-server`
and running the test causes error above. Applying the fix resolves the
issue.

Signed-off-by: Yakov Gusakov <[email protected]>
ranshid pushed a commit that referenced this pull request Oct 10, 2025
@hpatro
Fix memory leak with CLIENT LIST/KILL duplicate filters (valkey-io#2362)
155b0bb 
With valkey-io#1401, we introduced additional filters to CLIENT LIST/KILL
subcommand. The intended behavior was to pick the last value of the
filter. However, we introduced memory leak for all the preceding
filters.

Before this change:
```
> CLIENT LIST IP 127.0.0.1 IP 127.0.0.1
id=4 addr=127.0.0.1:37866 laddr=127.0.0.1:6379 fd=10 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=21 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=16989 events=r cmd=client|list user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=49 tot-net-out=0 tot-cmds=0
```
Leak:
```
Direct leak of 11 byte(s) in 1 object(s) allocated from:
    #0 0x7f2901aa557d in malloc (/lib64/libasan.so.4+0xd857d)
    #1 0x76db76 in ztrymalloc_usable_internal /workplace/harkrisp/valkey/src/zmalloc.c:156
    #2 0x76db76 in zmalloc_usable /workplace/harkrisp/valkey/src/zmalloc.c:200
    #3 0x4c4121 in _sdsnewlen.constprop.230 /workplace/harkrisp/valkey/src/sds.c:113
    #4 0x4dc456 in parseClientFiltersOrReply.constprop.63 /workplace/harkrisp/valkey/src/networking.c:4264
    #5 0x4bb9f7 in clientListCommand /workplace/harkrisp/valkey/src/networking.c:4600
    #6 0x641159 in call /workplace/harkrisp/valkey/src/server.c:3772
    #7 0x6431a6 in processCommand /workplace/harkrisp/valkey/src/server.c:4434
    valkey-io#8 0x4bfa9b in processCommandAndResetClient /workplace/harkrisp/valkey/src/networking.c:3571
    valkey-io#9 0x4bfa9b in processInputBuffer /workplace/harkrisp/valkey/src/networking.c:3702
    valkey-io#10 0x4bffa3 in readQueryFromClient /workplace/harkrisp/valkey/src/networking.c:3812
    valkey-io#11 0x481015 in callHandler /workplace/harkrisp/valkey/src/connhelpers.h:79
    valkey-io#12 0x481015 in connSocketEventHandler.lto_priv.394 /workplace/harkrisp/valkey/src/socket.c:301
    valkey-io#13 0x7d3fb3 in aeProcessEvents /workplace/harkrisp/valkey/src/ae.c:486
    valkey-io#14 0x7d4d44 in aeMain /workplace/harkrisp/valkey/src/ae.c:543
    valkey-io#15 0x453925 in main /workplace/harkrisp/valkey/src/server.c:7319
    valkey-io#16 0x7f2900cd7139 in __libc_start_main (/lib64/libc.so.6+0x21139)
```

Note: For filter ID / NOT-ID we group all the option and perform
filtering whereas for remaining filters we only pick the last filter
option.

---------

Signed-off-by: Harkrishn Patro <[email protected]>
ranshid pushed a commit that referenced this pull request Nov 26, 2025
@hpatro @madolson
Fix memory leak with CLIENT LIST/KILL duplicate filters (valkey-io#2362)
010fa64 
With valkey-io#1401, we introduced additional filters to CLIENT LIST/KILL
subcommand. The intended behavior was to pick the last value of the
filter. However, we introduced memory leak for all the preceding
filters.

Before this change:
```
> CLIENT LIST IP 127.0.0.1 IP 127.0.0.1
id=4 addr=127.0.0.1:37866 laddr=127.0.0.1:6379 fd=10 name= age=0 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=21 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=16989 events=r cmd=client|list user=default redir=-1 resp=2 lib-name= lib-ver= tot-net-in=49 tot-net-out=0 tot-cmds=0
```
Leak:
```
Direct leak of 11 byte(s) in 1 object(s) allocated from:
    #0 0x7f2901aa557d in malloc (/lib64/libasan.so.4+0xd857d)
    #1 0x76db76 in ztrymalloc_usable_internal /workplace/harkrisp/valkey/src/zmalloc.c:156
    #2 0x76db76 in zmalloc_usable /workplace/harkrisp/valkey/src/zmalloc.c:200
    #3 0x4c4121 in _sdsnewlen.constprop.230 /workplace/harkrisp/valkey/src/sds.c:113
    #4 0x4dc456 in parseClientFiltersOrReply.constprop.63 /workplace/harkrisp/valkey/src/networking.c:4264
    #5 0x4bb9f7 in clientListCommand /workplace/harkrisp/valkey/src/networking.c:4600
    #6 0x641159 in call /workplace/harkrisp/valkey/src/server.c:3772
    #7 0x6431a6 in processCommand /workplace/harkrisp/valkey/src/server.c:4434
    valkey-io#8 0x4bfa9b in processCommandAndResetClient /workplace/harkrisp/valkey/src/networking.c:3571
    valkey-io#9 0x4bfa9b in processInputBuffer /workplace/harkrisp/valkey/src/networking.c:3702
    valkey-io#10 0x4bffa3 in readQueryFromClient /workplace/harkrisp/valkey/src/networking.c:3812
    valkey-io#11 0x481015 in callHandler /workplace/harkrisp/valkey/src/connhelpers.h:79
    valkey-io#12 0x481015 in connSocketEventHandler.lto_priv.394 /workplace/harkrisp/valkey/src/socket.c:301
    valkey-io#13 0x7d3fb3 in aeProcessEvents /workplace/harkrisp/valkey/src/ae.c:486
    valkey-io#14 0x7d4d44 in aeMain /workplace/harkrisp/valkey/src/ae.c:543
    valkey-io#15 0x453925 in main /workplace/harkrisp/valkey/src/server.c:7319
    valkey-io#16 0x7f2900cd7139 in __libc_start_main (/lib64/libc.so.6+0x21139)
```

Note: For filter ID / NOT-ID we group all the option and perform
filtering whereas for remaining filters we only pick the last filter
option.

---------

Signed-off-by: Harkrishn Patro <[email protected]>
(cherry picked from commit 155b0bb)
Signed-off-by: cherukum-amazon <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@xbasel @ranshid