Skip to content

[CRASH] Segfault in expireScanCallback during activeExpireCycle (Valkey 8.1.3) #2669

@sergeij

Description

@sergeij

Crash report

=== VALKEY BUG REPORT START: Cut & paste starting from here ===
1:M 01 Oct 2025 03:18:36.984 # valkey 8.1.3 crashed by signal: 11, si_code: 1
1:M 01 Oct 2025 03:18:36.984 # Accessing address: 0x4
1:M 01 Oct 2025 03:18:36.984 # Crashed running the instruction at: 0x585ed780fb7d

------ STACK TRACE ------
EIP:
valkey-server *:6379(expireScanCallback+0xd)[0x585ed780fb7d]

1 valkey-server *
/lib/x86_64-linux-gnu/libc.so.6(+0x3c050)[0x7b02ce2c1050]
valkey-server *:6379(expireScanCallback+0xd)[0x585ed780fb7d]
valkey-server *:6379(activeExpireCycle+0xb84)[0x585ed7810824]
valkey-server *:6379(databasesCron+0x4c2)[0x585ed76e6632]
valkey-server *:6379(serverCron+0x74b)[0x585ed76e721b]
valkey-server *:6379(+0x8b0c4)[0x585ed76d00c4]
valkey-server *:6379(aeMain+0x130)[0x585ed76da1b0]
valkey-server *:6379(main+0x57a)[0x585ed76ce6da]
/lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7b02ce2ac24a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7b02ce2ac305]
valkey-server *:6379(_start+0x21)[0x585ed76cffa1]

37 bio_aof
/lib/x86_64-linux-gnu/libc.so.6(+0x85f16)[0x7b02ce30af16]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x1e8)[0x7b02ce30d5d8]
valkey-server *:6379(bioProcessBackgroundJobs+0x197)[0x585ed77b9087]
/lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7b02ce30e1f5]
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x40)[0x7b02ce38db00]

36 bio_close_file
/lib/x86_64-linux-gnu/libc.so.6(+0x85f16)[0x7b02ce30af16]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x1e8)[0x7b02ce30d5d8]
valkey-server *:6379(bioProcessBackgroundJobs+0x197)[0x585ed77b9087]
/lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7b02ce30e1f5]
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x40)[0x7b02ce38db00]

38 bio_lazy_free
/lib/x86_64-linux-gnu/libc.so.6(+0x85f16)[0x7b02ce30af16]
/lib/x86_64-linux-gnu/libc.so.6(pthread_cond_wait+0x1e8)[0x7b02ce30d5d8]
valkey-server *:6379(bioProcessBackgroundJobs+0x197)[0x585ed77b9087]
/lib/x86_64-linux-gnu/libc.so.6(+0x891f5)[0x7b02ce30e1f5]
/lib/x86_64-linux-gnu/libc.so.6(__clone+0x40)[0x7b02ce38db00]

4/4 expected stacktraces.

------ STACK TRACE DONE ------

------ REGISTERS ------
1:M 01 Oct 2025 03:18:36.992 # 
RAX:0000000000000005 RBX:00007b02bce2a740
RCX:0000000000000002 RDX:0000000000000004
RDI:00007ffdb51b2960 RSI:0000000000000000
RBP:00007ffdb51b2860 RSP:00007ffdb51b2840
R8 :fffffffffffff550 R9 :00007b02cde00c70
R10:00007b02ce280768 R11:00000bf15e20ad68
R12:0000000000000002 R13:0000000000000001
R14:00007b02cde47480 R15:0000000000000001
RIP:0000585ed780fb7d EFL:0000000000010202
CSGSFS:002b000000000033
1:M 01 Oct 2025 03:18:36.992 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.

------ INFO OUTPUT ------
# Server
redis_version:7.2.4
server_name:valkey
valkey_version:8.1.3
valkey_release_stage:ga
redis_git_sha1:00000000
redis_git_dirty:1
redis_build_id:7a72b458650d7456
server_mode:standalone
os:Linux 6.8.0-1030-aws x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
gcc_version:12.2.0
process_id:1
process_supervised:no
run_id:1a2bcb7aaa706ab39fff399e069f0fd307891534
tcp_port:6379
server_time_usec:1759288716984463
uptime_in_seconds:942904
uptime_in_days:10
hz:10
configured_hz:10
clients_hz:10
lru_clock:14458252
executable:/valkey-server
config_file:
io_threads_active:0
availability_zone:
listener0:name=tcp,bind=*,bind=-::*,port=6379

# Clients
connected_clients:66
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
pubsub_clients:3
watching_clients:0
clients_in_timeout_table:0
total_watched_keys:0
total_blocking_keys:0
total_blocking_keys_on_nokey:0
paused_reason:none
paused_actions:none
paused_timeout_milliseconds:0

# Memory
used_memory:1550396168
used_memory_human:1.44G
used_memory_rss:1599864832
used_memory_rss_human:1.49G
used_memory_peak:1585515560
used_memory_peak_human:1.48G
used_memory_peak_perc:97.78%
used_memory_overhead:61912320
used_memory_startup:889376
used_memory_dataset:1488483848
used_memory_dataset_perc:96.06%
allocator_allocated:1550770928
allocator_active:1567567872
allocator_resident:1595318272
allocator_muzzy:0
total_system_memory:8158130176
total_system_memory_human:7.60G
used_memory_lua:32768
used_memory_vm_eval:32768
used_memory_lua_human:32.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:33792
used_memory_vm_total:66560
used_memory_vm_total_human:65.00K
used_memory_functions:224
used_memory_scripts:224
used_memory_scripts_human:224B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:16796944
allocator_rss_ratio:1.02
allocator_rss_bytes:27750400
rss_overhead_ratio:1.00
rss_overhead_bytes:4546560
mem_fragmentation_ratio:1.03
mem_fragmentation_bytes:49470680
mem_not_counted_for_evict:7880
mem_replication_backlog:10485864
mem_total_replication_buffers:10532872
mem_clients_slaves:47112
mem_clients_normal:254504
mem_cluster_links:0
mem_aof_buffer:1792
mem_allocator:jemalloc-5.3.0
mem_overhead_db_hashtable_rehashing:0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:28

# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:36199211
rdb_bgsave_in_progress:0
rdb_last_save_time:1758345812
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:7
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:31232000
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:1065884
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:7
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:101
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:33730560
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:1095799679
aof_base_size:776609932
aof_pending_rewrite:0
aof_buffer_length:62
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats
total_connections_received:100643307
total_commands_processed:563943657
instantaneous_ops_per_sec:201
total_net_input_bytes:99257665731
total_net_output_bytes:13116755001384
total_net_repl_input_bytes:0
total_net_repl_output_bytes:157817942036
instantaneous_input_kbps:10.48
instantaneous_output_kbps:8250.38
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:6.37
rejected_connections:0
sync_full:2
sync_partial_ok:0
sync_partial_err:2
expired_keys:10342045
expired_stale_perc:1.81
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:278448
evicted_keys:0
evicted_clients:0
evicted_scripts:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:264441795
keyspace_misses:7230414
pubsub_channels:1
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:27178
total_forks:102
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:37
dump_payload_sanitizations:0
total_reads_processed:664967825
total_writes_processed:650297847
io_threaded_reads_processed:0
io_threaded_writes_processed:0
io_threaded_freed_objects:0
io_threaded_accept_processed:0
io_threaded_poll_processed:0
io_threaded_total_prefetch_batches:0
io_threaded_total_prefetch_entries:0
client_query_buffer_limit_disconnections:0
client_output_buffer_limit_disconnections:0
reply_buffer_shrinks:3780332
reply_buffer_expands:1693108
eventloop_cycles:736841144
eventloop_duration_sum:29448101737
eventloop_duration_cmd_sum:2888655844
instantaneous_eventloop_cycles_per_sec:286
instantaneous_eventloop_duration_usec:71
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0

# Replication
role:master
connected_slaves:2
slave0:ip=valkey-node-1.default-valkey-headless.default.svc.cluster.local,port=6379,state=online,offset=78160824927,lag=0,type=replica
slave1:ip=valkey-node-2.default-valkey-headless.default.svc.cluster.local,port=6379,state=online,offset=78160826359,lag=0,type=replica
replicas_waiting_psync:0
master_failover_state:no-failover
master_replid:0c1b569de90bdada3f0ae6f24b30968609c71c30
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:78160829548
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:10485760
repl_backlog_first_byte_offset:78150337251
repl_backlog_histlen:10492298

# CPU
used_cpu_sys:22139.325850
used_cpu_user:11741.507513
used_cpu_sys_children:88.932730
used_cpu_user_children:626.560572
used_cpu_sys_main_thread:21982.617265
used_cpu_user_main_thread:11731.478573

# Modules

# Commandstats
cmdstat_zrangebyscore:calls=5636,usec=36935,usec_per_call=6.55,rejected_calls=0,failed_calls=0
cmdstat_auth:calls=100549017,usec=306931596,usec_per_call=3.05,rejected_calls=0,failed_calls=0
cmdstat_zrem:calls=36098,usec=74704,usec_per_call=2.07,rejected_calls=0,failed_calls=0
cmdstat_exists:calls=23825035,usec=32026493,usec_per_call=1.34,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=12335427,usec=117336965,usec_per_call=9.51,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=47332120,usec=25691261,usec_per_call=0.54,rejected_calls=27,failed_calls=0
cmdstat_client|setname:calls=15721,usec=13559,usec_per_call=0.86,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=1872036,usec=3272735,usec_per_call=1.75,rejected_calls=0,failed_calls=0
cmdstat_del:calls=729471,usec=3890541,usec_per_call=5.33,rejected_calls=0,failed_calls=0
cmdstat_setex:calls=23063240,usec=134444363,usec_per_call=5.83,rejected_calls=0,failed_calls=0
cmdstat_get:calls=60236243,usec=105673010,usec_per_call=1.75,rejected_calls=0,failed_calls=0
cmdstat_subscribe:calls=3,usec=12,usec_per_call=4.00,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=2,usec=56,usec_per_call=28.00,rejected_calls=10,failed_calls=0
cmdstat_select:calls=100156129,usec=42477714,usec_per_call=0.42,rejected_calls=0,failed_calls=0
cmdstat_publish:calls=1387493,usec=13140230,usec_per_call=9.47,rejected_calls=0,failed_calls=0
cmdstat_mget:calls=187604816,usec=2038415126,usec_per_call=10.87,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=15715,usec=2878569,usec_per_call=183.17,rejected_calls=0,failed_calls=0
cmdstat_info:calls=297451,usec=31391678,usec_per_call=105.54,rejected_calls=0,failed_calls=0
cmdstat_set:calls=4450095,usec=30890351,usec_per_call=6.94,rejected_calls=0,failed_calls=0
cmdstat_zrange:calls=479,usec=4431,usec_per_call=9.25,rejected_calls=0,failed_calls=0
cmdstat_slowlog|get:calls=15715,usec=48889,usec_per_call=3.11,rejected_calls=0,failed_calls=0
cmdstat_slowlog|len:calls=15715,usec=16626,usec_per_call=1.06,rejected_calls=0,failed_calls=0

# Errorstats
errorstat_LOADING:count=25
errorstat_NOAUTH:count=12

# Latencystats
latency_percentiles_usec_zrangebyscore:p50=6.015,p99=24.063,p99.9=40.191
latency_percentiles_usec_auth:p50=3.007,p99=9.023,p99.9=31.103
latency_percentiles_usec_zrem:p50=1.003,p99=16.063,p99.9=28.031
latency_percentiles_usec_exists:p50=1.003,p99=5.023,p99.9=19.071
latency_percentiles_usec_zadd:p50=9.023,p99=27.007,p99.9=48.127
latency_percentiles_usec_ping:p50=0.001,p99=2.007,p99.9=11.007
latency_percentiles_usec_client|setname:p50=1.003,p99=3.007,p99.9=15.039
latency_percentiles_usec_replconf:p50=2.007,p99=3.007,p99.9=23.039
latency_percentiles_usec_del:p50=5.023,p99=15.039,p99.9=35.071
latency_percentiles_usec_setex:p50=5.023,p99=18.047,p99.9=38.143
latency_percentiles_usec_get:p50=1.003,p99=7.007,p99.9=26.111
latency_percentiles_usec_subscribe:p50=3.007,p99=6.015,p99.9=6.015
latency_percentiles_usec_psync:p50=28.031,p99=28.031,p99.9=28.031
latency_percentiles_usec_select:p50=0.001,p99=2.007,p99.9=8.031
latency_percentiles_usec_publish:p50=9.023,p99=28.031,p99.9=60.159
latency_percentiles_usec_mget:p50=9.023,p99=39.167,p99.9=64.255
latency_percentiles_usec_config|get:p50=176.127,p99=264.191,p99.9=391.167
latency_percentiles_usec_info:p50=97.279,p99=243.711,p99.9=317.439
latency_percentiles_usec_set:p50=7.007,p99=20.095,p99.9=38.143
latency_percentiles_usec_zrange:p50=7.007,p99=43.007,p99.9=96.255
latency_percentiles_usec_slowlog|get:p50=3.007,p99=11.007,p99.9=27.007
latency_percentiles_usec_slowlog|len:p50=1.003,p99=3.007,p99.9=18.047

# Cluster
cluster_enabled:0

# Keyspace
db1:keys=310576,expires=225298,avg_ttl=299624220
db2:keys=2,expires=0,avg_ttl=3193
db3:keys=1,expires=0,avg_ttl=0
db4:keys=13401,expires=13401,avg_ttl=2713508
db5:keys=13,expires=12,avg_ttl=828195
db6:keys=1,expires=0,avg_ttl=0
db9:keys=8,expires=0,avg_ttl=0
db11:keys=784704,expires=784703,avg_ttl=1320255251
db12:keys=4,expires=4,avg_ttl=137119

------ CLIENT LIST OUTPUT ------
id=98455479 addr=10.30.5.83:41136 laddr=10.30.10.193:6379 fd=36 name=*redacted* age=33927 idle=16 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1297905 tot-net-out=1008156130 tot-cmds=14378
id=98456794 addr=10.30.14.215:43220 laddr=10.30.10.193:6379 fd=22 name=*redacted* age=33910 idle=43 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=768088 tot-net-out=16623 tot-cmds=2771
id=100643210 addr=10.30.1.134:57410 laddr=10.30.10.193:6379 fd=32 name=*redacted* age=2 idle=2 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=301 tot-net-out=280638 tot-cmds=6
id=96449597 addr=10.30.14.213:58210 laddr=10.30.10.193:6379 fd=62 name=*redacted* age=44934 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=3706881 tot-net-out=40101106 tot-cmds=112330
id=100624524 addr=10.30.14.213:46264 laddr=10.30.10.193:6379 fd=51 name=*redacted* age=346 idle=36 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=545 tot-net-out=561266 tot-cmds=10
id=100624525 addr=10.30.14.213:46276 laddr=10.30.10.193:6379 fd=55 name=*redacted* age=346 idle=36 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=4861 tot-net-out=106 tot-cmds=18
id=99621088 addr=10.30.9.20:60058 laddr=10.30.10.193:6379 fd=46 name=*redacted* age=21137 idle=118 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=31275 tot-net-out=311244 tot-cmds=948
id=100506992 addr=10.30.13.192:50432 laddr=10.30.10.193:6379 fd=27 name=*redacted* age=2005 idle=140 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=223467 tot-net-out=2254420 tot-cmds=6772
id=100506993 addr=10.30.6.63:53120 laddr=10.30.10.193:6379 fd=28 name=*redacted* age=2005 idle=140 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=223467 tot-net-out=2254420 tot-cmds=6772
id=100506994 addr=10.30.4.11:58654 laddr=10.30.10.193:6379 fd=35 name=*redacted* age=2005 idle=140 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=223467 tot-net-out=2254420 tot-cmds=6772
id=100643279 addr=10.30.1.140:47376 laddr=10.30.10.193:6379 fd=45 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=301 tot-net-out=280638 tot-cmds=6
id=100506995 addr=10.30.14.214:56780 laddr=10.30.10.193:6379 fd=50 name=*redacted* age=2005 idle=140 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=223467 tot-net-out=2254420 tot-cmds=6772
id=98693391 addr=10.30.6.50:55208 laddr=10.30.10.193:6379 fd=38 name=*redacted* age=31186 idle=43 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=107043 tot-net-out=891560 tot-cmds=3244
id=98511471 addr=10.30.4.12:56340 laddr=10.30.10.193:6379 fd=37 name=*redacted* age=33336 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1022554 tot-net-out=10096245 tot-cmds=30975
id=8 addr=10.30.10.193:60256 laddr=10.30.10.193:6379 fd=14 name=*redacted* age=942903 idle=0 flags=P capa= db=0 sub=1 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=277 obl=0 oll=0 omem=0 tot-mem=22200 events=r cmd=subscribe user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=138 tot-net-out=384335065 tot-cmds=3
id=9 addr=10.30.28.244:40732 laddr=10.30.10.193:6379 fd=15 name=*redacted* age=942903 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23168 events=r cmd=publish user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=142116280 tot-net-out=659336186 tot-cmds=1463944
id=10 addr=10.30.28.244:40736 laddr=10.30.10.193:6379 fd=16 name=*redacted* age=942903 idle=0 flags=P capa= db=0 sub=1 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=277 obl=0 oll=0 omem=0 tot-mem=22200 events=r cmd=subscribe user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=138 tot-net-out=384334788 tot-cmds=3
id=21 addr=10.30.32.25:34762 laddr=10.30.10.193:6379 fd=18 name=*redacted* age=942898 idle=0 flags=S capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=22168 events=r cmd=replconf user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=67360749 tot-net-out=78160826669 tot-cmds=935930
id=98458124 addr=10.30.5.81:43246 laddr=10.30.10.193:6379 fd=54 name=*redacted* age=33892 idle=237 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=390923 tot-net-out=8115 tot-cmds=1353
id=100643186 addr=10.30.1.137:46276 laddr=10.30.10.193:6379 fd=24 name=*redacted* age=3 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=606 tot-net-out=631423 tot-cmds=11
id=98455307 addr=10.30.5.81:39958 laddr=10.30.10.193:6379 fd=33 name=*redacted* age=33929 idle=118 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=58467 tot-net-out=582340 tot-cmds=1772
id=100643252 addr=10.30.14.69:56896 laddr=10.30.10.193:6379 fd=21 name=*redacted* age=1 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=362 tot-net-out=350795 tot-cmds=7
id=100643219 addr=10.30.1.134:57422 laddr=10.30.10.193:6379 fd=41 name=*redacted* age=2 idle=1 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=301 tot-net-out=280638 tot-cmds=6
id=100643287 addr=10.30.15.177:53022 laddr=10.30.10.193:6379 fd=19 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=240 tot-net-out=210481 tot-cmds=5
id=100514027 addr=10.30.9.17:40746 laddr=10.30.10.193:6379 fd=34 name=*redacted* age=1848 idle=141 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=221157 tot-net-out=2204310 tot-cmds=6702
id=100514028 addr=10.30.4.9:38190 laddr=10.30.10.193:6379 fd=48 name=*redacted* age=1848 idle=141 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=221157 tot-net-out=2204310 tot-cmds=6702
id=100514029 addr=10.30.4.8:52034 laddr=10.30.10.193:6379 fd=73 name=*redacted* age=1848 idle=141 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=221157 tot-net-out=2204310 tot-cmds=6702
id=100514030 addr=10.30.14.216:59594 laddr=10.30.10.193:6379 fd=75 name=*redacted* age=1848 idle=141 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=221157 tot-net-out=2204310 tot-cmds=6702
id=100638355 addr=10.30.6.51:59220 laddr=10.30.10.193:6379 fd=57 name=*redacted* age=118 idle=118 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=118 tot-net-out=70167 tot-cmds=3
id=100643289 addr=10.30.1.140:47380 laddr=10.30.10.193:6379 fd=39 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=179 tot-net-out=140324 tot-cmds=4
id=100638356 addr=10.30.6.51:59230 laddr=10.30.10.193:6379 fd=61 name=*redacted* age=118 idle=118 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=633 tot-net-out=22 tot-cmds=4
id=100514031 addr=10.30.14.216:59600 laddr=10.30.10.193:6379 fd=76 name=*redacted* age=1848 idle=141 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=83566 tot-net-out=96044943 tot-cmds=1371
id=100514032 addr=10.30.14.216:59602 laddr=10.30.10.193:6379 fd=30 name=*redacted* age=1848 idle=141 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=785475 tot-net-out=16438 tot-cmds=2740
id=100593776 addr=10.30.13.198:50834 laddr=10.30.10.193:6379 fd=20 name=*redacted* age=710 idle=111 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=362 tot-net-out=350795 tot-cmds=7
id=100593779 addr=10.30.13.198:50836 laddr=10.30.10.193:6379 fd=58 name=*redacted* age=710 idle=111 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=2837 tot-net-out=70 tot-cmds=12
id=98511511 addr=10.30.4.10:43512 laddr=10.30.10.193:6379 fd=70 name=*redacted* age=33335 idle=43 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=115095 tot-net-out=958660 tot-cmds=3488
id=98455314 addr=10.30.5.81:39968 laddr=10.30.10.193:6379 fd=47 name=*redacted* age=33929 idle=237 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=41293 tot-net-out=47426142 tot-cmds=678
id=5 addr=10.30.32.25:41624 laddr=10.30.10.193:6379 fd=11 name=*redacted* age=942903 idle=0 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=2048 rbp=1024 obl=0 oll=0 omem=0 tot-mem=23168 events=r cmd=publish user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=142120779 tot-net-out=659336899 tot-cmds=1463946
id=100642107 addr=10.30.15.177:51434 laddr=10.30.10.193:6379 fd=59 name=*redacted* age=27 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=8192 rbp=4096 obl=0 oll=0 omem=0 tot-mem=8832 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=2009 tot-net-out=2245034 tot-cmds=34
id=96442763 addr=10.30.14.223:43860 laddr=10.30.10.193:6379 fd=90 name=*redacted* age=44954 idle=16 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=30650456 tot-net-out=614199 tot-cmds=102367
id=98456556 addr=10.30.5.82:60966 laddr=10.30.10.193:6379 fd=74 name=*redacted* age=33913 idle=118 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=58467 tot-net-out=582340 tot-cmds=1772
id=98684853 addr=10.30.13.195:43178 laddr=10.30.10.193:6379 fd=99 name=*redacted* age=31310 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=973384 tot-net-out=9610505 tot-cmds=29485
id=100643298 addr=10.30.14.69:33664 laddr=10.30.10.193:6379 fd=40 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=240 tot-net-out=210481 tot-cmds=5
id=98693304 addr=10.30.6.51:43200 laddr=10.30.10.193:6379 fd=29 name=*redacted* age=31188 idle=118 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=52131 tot-net-out=519172 tot-cmds=1580
id=20 addr=10.30.28.244:40768 laddr=10.30.10.193:6379 fd=17 name=*redacted* age=942898 idle=0 flags=S capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=1 omem=20504 tot-mem=22168 events=r cmd=replconf user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=67370634 tot-net-out=78160826669 tot-cmds=936070
id=6 addr=10.30.32.25:41634 laddr=10.30.10.193:6379 fd=12 name=*redacted* age=942903 idle=0 flags=P capa= db=0 sub=1 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=277 obl=0 oll=0 omem=0 tot-mem=22200 events=r cmd=subscribe user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=138 tot-net-out=384335342 tot-cmds=3
id=99621061 addr=10.30.6.57:54516 laddr=10.30.10.193:6379 fd=43 name=*redacted* age=21138 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1005303 tot-net-out=10874944 tot-cmds=30464
id=100514043 addr=10.30.6.63:41208 laddr=10.30.10.193:6379 fd=82 name=*redacted* age=1848 idle=140 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=83261 tot-net-out=95694158 tot-cmds=1366
id=100514044 addr=10.30.6.63:41216 laddr=10.30.10.193:6379 fd=83 name=*redacted* age=1848 idle=140 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=804338 tot-net-out=16378 tot-cmds=2730
id=100643302 addr=10.30.1.137:46324 laddr=10.30.10.193:6379 fd=10 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=240 tot-net-out=210481 tot-cmds=5
id=100643304 addr=10.30.13.186:54608 laddr=10.30.10.193:6379 fd=26 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=301 tot-net-out=280638 tot-cmds=6
id=100643305 addr=10.30.14.219:54806 laddr=10.30.10.193:6379 fd=44 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=118 tot-net-out=70167 tot-cmds=3
id=98455837 addr=10.30.14.215:35764 laddr=10.30.10.193:6379 fd=49 name=*redacted* age=33923 idle=43 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=115953 tot-net-out=965810 tot-cmds=3514
id=98455839 addr=10.30.14.215:35772 laddr=10.30.10.193:6379 fd=53 name=*redacted* age=33923 idle=43 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=84542 tot-net-out=97167455 tot-cmds=1387
id=99619327 addr=10.30.14.222:52988 laddr=10.30.10.193:6379 fd=69 name=*redacted* age=21162 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=603825 tot-net-out=5964506 tot-cmds=18298
id=7 addr=10.30.10.193:60244 laddr=10.30.10.193:6379 fd=13 name=*redacted* age=942903 idle=1 flags=N capa= db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=7 obl=0 oll=0 omem=0 tot-mem=22144 events=r cmd=ping user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=142170903 tot-net-out=659362992 tot-cmds=1464746
id=96442372 addr=10.30.14.223:49336 laddr=10.30.10.193:6379 fd=63 name=*redacted* age=44956 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=3715131 tot-net-out=40190356 tot-cmds=112580
id=98455529 addr=10.30.5.83:41142 laddr=10.30.10.193:6379 fd=25 name=*redacted* age=33926 idle=16 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=8738290 tot-net-out=172371 tot-cmds=28729
id=100643307 addr=10.30.14.219:54814 laddr=10.30.10.193:6379 fd=31 name=*redacted* age=0 idle=0 flags=N capa= db=1 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=109 tot-net-out=615 tot-cmds=3
id=100643308 addr=10.30.32.40:52652 laddr=10.30.10.193:6379 fd=60 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=118 tot-net-out=70167 tot-cmds=3
id=96442376 addr=10.30.14.223:49344 laddr=10.30.10.193:6379 fd=77 name=*redacted* age=44956 idle=16 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=3824960 tot-net-out=3591547361 tot-cmds=51205
id=98456574 addr=10.30.3.225:45880 laddr=10.30.10.193:6379 fd=85 name=*redacted* age=33913 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1844229 tot-net-out=19950598 tot-cmds=55886
id=100624519 addr=10.30.13.195:47872 laddr=10.30.10.193:6379 fd=81 name=*redacted* age=346 idle=36 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=423 tot-net-out=420952 tot-cmds=8
id=100624520 addr=10.30.13.195:47888 laddr=10.30.10.193:6379 fd=84 name=*redacted* age=346 idle=36 flags=N capa= db=4 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=set user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=3597 tot-net-out=82 tot-cmds=14
id=98685151 addr=10.30.13.198:36884 laddr=10.30.10.193:6379 fd=87 name=*redacted* age=31305 idle=43 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=107043 tot-net-out=891560 tot-cmds=3244
id=98455477 addr=10.30.5.83:41128 laddr=10.30.10.193:6379 fd=56 name=*redacted* age=33927 idle=16 flags=N capa= db=5 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1664 events=r cmd=get user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=1040135 tot-net-out=10265128 tot-cmds=31496
id=100643309 addr=10.30.32.40:52654 laddr=10.30.10.193:6379 fd=52 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=mget user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=118 tot-net-out=70167 tot-cmds=3
id=100643310 addr=10.30.14.67:50748 laddr=10.30.10.193:6379 fd=23 name=*redacted* age=0 idle=0 flags=N capa= db=9 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17024 events=r cmd=select user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=57 tot-net-out=10 tot-cmds=2

------ MODULES INFO OUTPUT ------

------ CONFIG DEBUG OUTPUT ------
repl-diskless-sync yes
dual-channel-replication-enabled no
sanitize-dump-payload no
lazyfree-lazy-eviction yes
repl-diskless-load disabled
activedefrag no
lazyfree-lazy-user-flush yes
proto-max-bulk-len 512mb
replica-read-only yes
slave-read-only yes
client-query-buffer-limit 1gb
io-threads 1
import-mode no
lazyfree-lazy-server-del yes
debug-context ""
lazyfree-lazy-expire yes
list-compress-depth 0
lazyfree-lazy-user-del yes

------ FAST MEMORY TEST ------
1:M 01 Oct 2025 03:18:36.993 # Bio worker thread #0 terminated
1:M 01 Oct 2025 03:18:36.993 # Bio worker thread #1 terminated
1:M 01 Oct 2025 03:18:36.993 # Bio worker thread #2 terminated
*** Preparing to test memory region 585ed7a08000 (2355200 bytes)
*** Preparing to test memory region 585ed85e0000 (536576 bytes)
*** Preparing to test memory region 7b0247600000 (335544320 bytes)
*** Preparing to test memory region 7b025b800000 (8388608 bytes)
*** Preparing to test memory region 7b025c000000 (135168 bytes)
*** Preparing to test memory region 7b0260600000 (2097152 bytes)
*** Preparing to test memory region 7b0260a00000 (8388608 bytes)
*** Preparing to test memory region 7b0261200000 (2097152 bytes)
*** Preparing to test memory region 7b0261600000 (8388608 bytes)
*** Preparing to test memory region 7b0261e00000 (1746927616 bytes)
*** Preparing to test memory region 7b02ca100000 (13631488 bytes)
*** Preparing to test memory region 7b02caf00000 (9437184 bytes)
*** Preparing to test memory region 7b02cb9fd000 (8388608 bytes)
*** Preparing to test memory region 7b02cc1fe000 (8388608 bytes)
*** Preparing to test memory region 7b02cc9ff000 (8388608 bytes)
*** Preparing to test memory region 7b02cd200000 (8388608 bytes)
*** Preparing to test memory region 7b02cda00000 (8388608 bytes)
*** Preparing to test memory region 7b02ce280000 (20480 bytes)
*** Preparing to test memory region 7b02ce459000 (53248 bytes)
*** Preparing to test memory region 7b02ce8e9000 (12288 bytes)
*** Preparing to test memory region 7b02cea77000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: expireScanCallback (base: 0x585ed780fb70)
Module: valkey-server *:6379 (base 0x585ed7645000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x585ed780fb70 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:M 01 Oct 2025 03:18:41.504 # dump of function (hexdump of 141 bytes):
554889e541554154534883ec08f64604010f84f9000000488b4610488b57084889fb488b3f4989c44929d44885c00f88dc0000004839d07c274d85e47e084c0163208343280148834310014883c4085b415c415d5dc3662e0f1f840000000000e8cbfdffff85c074d04c8d2d40c220004883431801418b8df005000085c975b9e83b89feffe836e6edff418b95

=== VALKEY BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash by opening an issue on github:

           https://github.com/valkey-io/valkey/issues

  If a module was involved, please open in the module's repo instead.

  Suspect RAM error? Use valkey-server --test-memory to verify it.

  Some other issues could be detected by valkey-server --check-system

Additional information

Environment:
- Valkey version: 8.1.3 (based on Redis 7.2.4)
- Mode: Standalone, AOF enabled
- Uptime before crash: ~10 days
- Memory: ~5.5 GB used, fragmentation ratio ~1.01
- Large number of expiring keys: db11 (~785k), db1 (~225k)
- Config includes lazyfree-lazy-expire yes

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions