KeyDB 5.3.2 crashed by signal: 11

[botzill]

Describe the bug
I do run keydb on kuberentes, single pod, latest tag of keydb. At some point it crashes.

Can this be because I set server-threads 4 ?

** Log Files **
=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1:M 19 Mar 2020 15:50:26.250 # KeyDB 5.3.2 crashed by signal: 11
1:M 19 Mar 2020 15:50:26.250 # Crashed running the instruction at: 0x7f32ce2078f0
1:M 19 Mar 2020 15:50:26.250 # Accessing address: (nil)
1:M 19 Mar 2020 15:50:26.250 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7f32ce2078f0]

Backtrace:
keydb-server: semiorderedset.h:324: bool semiorderedset<T, T_KEY, MEMMOVE_SAFE>::enumerate_bucket(semiorderedset<T, T_KEY, MEMMOVE_SAFE>::setiter&, const T_MAX&, T_VISITOR&, long long int*) [with T_VISITOR = activeExpireCycle(int)::<lambda(expireEntry&)>; T_MAX = long long int; T = expireEntry; T_KEY = const char*; bool MEMMOVE_SAFE = true]: Assertion `(itr.idxSecondary+1) >= vec.size() || static_cast<T_MAX>(vec[itr.idxSecondary]) <= static_cast<T_MAX>(vec[itr.idxSecondary+1])' failed.
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x5643fdd2a55b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x5643fdd2acf9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f32ce5ca890]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7f32ce2078f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x3039a) [0x7f32ce1f739a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30412) [0x7f32ce1f7412]
keydb-server *:6379(activeExpireCycle(int)+0x513) [0x5643fdd63923]
keydb-server :6379(databasesCron()+0x38) [0x5643fdccebd8]
keydb-server :6379(serverCron(aeEventLoop, long long, void)+0x28c) [0x5643fdcd1b3c]
keydb-server *:6379(aeProcessEvents+0x25b) [0x5643fdcc964b]
keydb-server *:6379(aeMain+0x45) [0x5643fdcc9925]
keydb-server :6379(workerThreadMain(void)+0x97) [0x5643fdccde87]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f32ce5bf6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f32ce2e888f]

------ INFO OUTPUT ------

Server

redis_version:5.3.2
redis_git_sha1:e523afa7
redis_git_dirty:0
redis_build_id:642427f2cdc0fe4e
redis_mode:standalone
os:Linux 4.19.0-0.bpo.6-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.4.0
process_id:1
run_id:79ce13fd32099ac66d3e4a422a78fee2a9c6a8e3
tcp_port:6379
uptime_in_seconds:1465
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7574722
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

Clients

connected_clients:73
client_recent_max_input_buffer:13530
client_recent_max_output_buffer:0
blocked_clients:1
current_client_thread:0
thread_0_clients:50
thread_1_clients:23
thread_2_clients:0
thread_3_clients:0

Memory

used_memory:23503352
used_memory_human:22.41M
used_memory_rss:58351616
used_memory_rss_human:55.65M
used_memory_peak:23565384
used_memory_peak_human:22.47M
used_memory_peak_perc:99.74%
used_memory_overhead:7544302
used_memory_startup:5457504
used_memory_dataset:15959050
used_memory_dataset_perc:88.44%
allocator_allocated:24265400
allocator_active:25518080
allocator_resident:29024256
total_system_memory:33732517888
total_system_memory_human:31.42G
used_memory_lua:95232
used_memory_lua_human:93.00K
used_memory_scripts:16712
used_memory_scripts_human:16.32K
number_of_cached_scripts:9
maxmemory:0
maxmemory_human:0B
maxmemory_policy:allkeys-lfu
allocator_frag_ratio:1.05
allocator_frag_bytes:1252680
allocator_rss_ratio:1.14
allocator_rss_bytes:3506176
rss_overhead_ratio:2.01
rss_overhead_bytes:29327360
mem_fragmentation_ratio:2.48
mem_fragmentation_bytes:34808336
mem_not_counted_for_evict:3424
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:1334814
mem_aof_buffer:3424
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:119604
rdb_bgsave_in_progress:0
rdb_last_save_time:1584631561
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:23271343
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Stats

total_connections_received:116
total_commands_processed:412027
instantaneous_ops_per_sec:492
total_net_input_bytes:48689960
total_net_output_bytes:32138724
instantaneous_input_kbps:53.99
instantaneous_output_kbps:45.14
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:481
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:225764
keyspace_misses:17156
pubsub_channels:3
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:c1efbf7faf438b91635abf4b992f698640dff6ac
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:25.256737
used_cpu_user:17.486800
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:2

Commandstats

cmdstat_exec:calls=510,usec=79679,usec_per_call=156.23
cmdstat_zrem:calls=2218,usec=6650,usec_per_call=3.00
cmdstat_zadd:calls=1799,usec=27740,usec_per_call=15.42
cmdstat_set:calls=9779,usec=57373,usec_per_call=5.87
cmdstat_hincrbyfloat:calls=43087,usec=1037526,usec_per_call=24.08
cmdstat_client:calls=3,usec=10,usec_per_call=3.33
cmdstat_lrange:calls=707,usec=18132,usec_per_call=25.65
cmdstat_info:calls=160,usec=18688,usec_per_call=116.80
cmdstat_keys:calls=1,usec=9,usec_per_call=9.00
cmdstat_zrangebyscore:calls=1537,usec=16622,usec_per_call=10.81
cmdstat_zcard:calls=9,usec=6,usec_per_call=0.67
cmdstat_subscribe:calls=3,usec=14,usec_per_call=4.67
cmdstat_zrange:calls=1537,usec=12334,usec_per_call=8.02
cmdstat_script:calls=441,usec=2487,usec_per_call=5.64
cmdstat_zcount:calls=1,usec=4,usec_per_call=4.00
cmdstat_zscore:calls=951,usec=2862,usec_per_call=3.01
cmdstat_publish:calls=17361,usec=29275,usec_per_call=1.69
cmdstat_hset:calls=16469,usec=112627,usec_per_call=6.84
cmdstat_incr:calls=12240,usec=76940,usec_per_call=6.29
cmdstat_smembers:calls=49,usec=561,usec_per_call=11.45
cmdstat_zrevrange:calls=49,usec=167,usec_per_call=3.41
cmdstat_exists:calls=14321,usec=24028,usec_per_call=1.68
cmdstat_linsert:calls=1,usec=2,usec_per_call=2.00
cmdstat_lindex:calls=1,usec=2,usec_per_call=2.00
cmdstat_evalsha:calls=24644,usec=2903510,usec_per_call=117.82
cmdstat_sadd:calls=41,usec=738,usec_per_call=18.00
cmdstat_hmget:calls=111952,usec=664773,usec_per_call=5.94
cmdstat_hdel:calls=118,usec=1032,usec_per_call=8.75
cmdstat_auth:calls=116,usec=469,usec_per_call=4.04
cmdstat_hmset:calls=13681,usec=196533,usec_per_call=14.37
cmdstat_eval:calls=48,usec=3445,usec_per_call=71.77
cmdstat_brpoplpush:calls=213,usec=2033,usec_per_call=9.54
cmdstat_command:calls=1,usec=3559,usec_per_call=3559.00
cmdstat_hgetall:calls=52853,usec=696188,usec_per_call=13.17
cmdstat_llen:calls=10,usec=16,usec_per_call=1.60
cmdstat_del:calls=1033,usec=3756,usec_per_call=3.64
cmdstat_rpoplpush:calls=1441,usec=13264,usec_per_call=9.20
cmdstat_srem:calls=8295,usec=33600,usec_per_call=4.05
cmdstat_hexists:calls=45771,usec=360174,usec_per_call=7.87
cmdstat_lpush:calls=12541,usec=81918,usec_per_call=6.53
cmdstat_get:calls=10050,usec=72066,usec_per_call=7.17
cmdstat_lrem:calls=2321,usec=32112,usec_per_call=13.84
cmdstat_hget:calls=3122,usec=28393,usec_per_call=9.09
cmdstat_expiremember:calls=32,usec=384,usec_per_call=12.00
cmdstat_multi:calls=510,usec=739,usec_per_call=1.45

Cluster

cluster_enabled:0

Keyspace

db0:keys=12398,expires=129,avg_ttl=14612

------ CLIENT LIST OUTPUT ------
id=34 addr=10.244.5.35:56698 fd=98 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=99 addr=10.244.5.35:45184 fd=147 name= age=355 idle=355 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=64 addr=10.244.5.35:35204 fd=120 name= age=915 idle=915 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=66 addr=10.244.5.35:35208 fd=122 name= age=915 idle=915 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=36 addr=10.244.5.35:56706 fd=99 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=1259 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=37 addr=10.244.5.35:56708 fd=100 name= age=1259 idle=1228 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=13524 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=38 addr=10.244.5.35:56710 fd=101 name= age=1259 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=39 addr=10.244.5.35:56720 fd=102 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=41 addr=10.244.5.35:56724 fd=104 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=71 addr=10.244.5.35:36694 fd=124 name= age=836 idle=836 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=42 addr=10.244.5.35:56726 fd=103 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=28 addr=10.244.5.35:56464 fd=93 name= age=1277 idle=1277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=89 addr=10.244.5.35:44490 fd=140 name= age=393 idle=393 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=45 addr=10.244.5.35:56946 fd=107 name= age=1246 idle=1246 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=70 addr=10.244.5.35:36692 fd=125 name= age=836 idle=836 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=46 addr=10.244.5.35:56948 fd=106 name= age=1246 idle=1246 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=60 addr=10.244.5.35:34428 fd=117 name= age=956 idle=956 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=87 addr=10.244.5.35:44486 fd=138 name= age=393 idle=393 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=62 addr=10.244.5.35:34432 fd=119 name= age=956 idle=956 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=79 addr=10.244.5.35:36818 fd=132 name= age=829 idle=829 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=81 addr=10.244.5.35:36822 fd=134 name= age=829 idle=829 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=82 addr=10.244.5.35:36826 fd=133 name= age=829 idle=829 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=103 addr=10.244.5.35:45412 fd=150 name= age=343 idle=343 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=114 addr=10.244.5.35:45596 fd=159 name= age=333 idle=333 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=116 addr=10.244.5.35:45600 fd=161 name= age=333 idle=333 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=117 addr=10.244.5.35:45602 fd=160 name= age=333 idle=333 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=105 addr=10.244.5.35:45416 fd=152 name= age=343 idle=343 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=106 addr=10.244.5.35:45418 fd=151 name= age=343 idle=343 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=63 addr=10.244.5.35:34434 fd=118 name= age=956 idle=956 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=68 addr=10.244.5.35:36684 fd=123 name= age=836 idle=836 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=35 addr=10.244.5.35:56700 fd=97 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=58 addr=10.244.5.35:59376 fd=115 name= age=1130 idle=1130 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=98 addr=10.244.5.35:44700 fd=145 name= age=381 idle=381 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=78 addr=10.244.5.35:36812 fd=131 name= age=830 idle=3 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=94 addr=10.244.5.35:44582 fd=142 name= age=388 idle=388 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=30 addr=10.244.5.35:56468 fd=95 name= age=1277 idle=1277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=31 addr=10.244.5.35:56470 fd=94 name= age=1277 idle=1277 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=32 addr=10.244.5.35:56694 fd=96 name= age=1259 idle=1259 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=76 addr=10.244.5.35:36808 fd=129 name=queue:MGZjODJjZjEtZmVhMS00NTMzLWFkYmYtOGYyYjMzOTFmZDJi age=830 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=77 addr=10.244.5.35:36810 fd=130 name= age=830 idle=664 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=2622 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=47 addr=10.244.5.35:56990 fd=108 name= age=1242 idle=1242 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=49 addr=10.244.5.35:56994 fd=110 name= age=1242 idle=1242 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=97 addr=10.244.5.35:44694 fd=146 name= age=381 idle=381 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=75 addr=10.244.5.35:36800 fd=128 name= age=830 idle=830 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=50 addr=10.244.5.35:56996 fd=109 name= age=1242 idle=1242 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=43 addr=10.244.5.35:56942 fd=105 name= age=1246 idle=1246 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=95 addr=10.244.5.35:44690 fd=144 name= age=381 idle=381 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=55 addr=10.244.5.35:59366 fd=114 name= age=1130 idle=1130 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=57 addr=10.244.5.35:59370 fd=116 name= age=1130 idle=1130 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=91 addr=10.244.5.35:44576 fd=141 name= age=388 idle=388 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=93 addr=10.244.5.35:44580 fd=143 name= age=388 idle=388 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=74 addr=10.244.5.35:36802 fd=127 name= age=830 idle=830 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=27 addr=10.244.5.35:56098 fd=92 name= age=1304 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hmget
id=83 addr=10.244.5.35:37010 fd=135 name= age=818 idle=818 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=108 addr=10.244.5.35:45422 fd=154 name= age=342 idle=5 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=2622 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=109 addr=10.244.5.35:45424 fd=155 name= age=342 idle=2 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=85 addr=10.244.5.35:37016 fd=137 name= age=818 idle=818 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=121 addr=10.244.5.35:45798 fd=163 name= age=322 idle=322 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=113 addr=10.244.5.35:45440 fd=157 name= age=342 idle=342 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=86 addr=10.244.5.35:37022 fd=136 name= age=818 idle=818 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=51 addr=10.244.5.35:59058 fd=111 name= age=1145 idle=1145 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=67 addr=10.244.5.35:35210 fd=121 name= age=915 idle=915 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=53 addr=10.244.5.35:59062 fd=113 name= age=1145 idle=1145 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=107 addr=10.244.5.35:45420 fd=153 name=queue:ZWNhZWEwOWYtYjYyZC00NzZkLTljNTQtMTFlYmE1ODRhMmZm age=342 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=112 addr=10.244.5.35:45438 fd=158 name= age=342 idle=342 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=90 addr=10.244.5.35:44496 fd=139 name= age=393 idle=392 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=101 addr=10.244.5.35:45188 fd=149 name= age=355 idle=355 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=102 addr=10.244.5.35:45192 fd=148 name= age=355 idle=355 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=72 addr=10.244.5.35:36796 fd=126 name= age=830 idle=830 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=118 addr=10.244.5.35:45792 fd=162 name= age=322 idle=322 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=120 addr=10.244.5.35:45796 fd=164 name= age=322 idle=322 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=110 addr=10.244.5.35:45430 fd=156 name= age=342 idle=342 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=54 addr=10.244.5.35:59066 fd=112 name= age=1145 idle=1145 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info

------ REGISTERS ------
1:M 19 Mar 2020 15:50:26.252 #
RAX:0000000000000004 RBX:0000000000000000
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000002 RSI:00007f32c647a0f0
RBP:00007f32ce37e7d8 RSP:00007f32c647a210
R8 :0000000000000000 R9 :00007f32c647a0f0
R10:0000000000000008 R11:0000000000000246
R12:00005643fde1d57f R13:00005643fde31e78
R14:0000000000000144 R15:8000000000000000
RIP:00007f32ce2078f0 EFL:0000000000010246
CSGSFS:002b000000000033
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21f) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21e) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21d) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21c) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21b) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a21a) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a219) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a218) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a217) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a216) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a215) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a214) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a213) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a212) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a211) -> 0000000000000000
1:M 19 Mar 2020 15:50:26.252 # (00007f32c647a210) -> 0000000000000020

------ FAST MEMORY TEST ------
1:M 19 Mar 2020 15:50:26.252 # Bio thread for job type #0 terminated
1:M 19 Mar 2020 15:50:26.252 # Bio thread for job type #1 terminated
1:M 19 Mar 2020 15:50:26.252 # Bio thread for job type #2 terminated
*** Preparing to test memory region 5643fe079000 (2260992 bytes)
*** Preparing to test memory region 5643ff411000 (323584 bytes)
*** Preparing to test memory region 7f32b4000000 (139264 bytes)
*** Preparing to test memory region 7f32bc000000 (327680 bytes)
*** Preparing to test memory region 7f32c1e00000 (33554432 bytes)
*** Preparing to test memory region 7f32c3fff000 (2097152 bytes)
*** Preparing to test memory region 7f32c4200000 (8388608 bytes)
*** Preparing to test memory region 7f32c4a00000 (2097152 bytes)
*** Preparing to test memory region 7f32c4c7b000 (8388608 bytes)
*** Preparing to test memory region 7f32c547c000 (8388608 bytes)
*** Preparing to test memory region 7f32c5c7d000 (8388608 bytes)
*** Preparing to test memory region 7f32c647e000 (8388608 bytes)
*** Preparing to test memory region 7f32c6c7f000 (8388608 bytes)
*** Preparing to test memory region 7f32c7480000 (8388608 bytes)
*** Preparing to test memory region 7f32c7c80000 (14155776 bytes)
*** Preparing to test memory region 7f32c8d53000 (188416 bytes)
*** Preparing to test memory region 7f32c9089000 (4096 bytes)
*** Preparing to test memory region 7f32c92d3000 (4096 bytes)
*** Preparing to test memory region 7f32c9d5f000 (4096 bytes)
*** Preparing to test memory region 7f32ca28e000 (4096 bytes)
*** Preparing to test memory region 7f32cabdd000 (4096 bytes)
*** Preparing to test memory region 7f32cb253000 (8192 bytes)
*** Preparing to test memory region 7f32cb895000 (4096 bytes)
*** Preparing to test memory region 7f32cc5bc000 (4096 bytes)
*** Preparing to test memory region 7f32ccfb6000 (8192 bytes)
*** Preparing to test memory region 7f32cd6cb000 (12288 bytes)
*** Preparing to test memory region 7f32ce5b4000 (16384 bytes)
*** Preparing to test memory region 7f32ce7d3000 (16384 bytes)
*** Preparing to test memory region 7f32cf112000 (16384 bytes)
*** Preparing to test memory region 7f32cfbb0000 (114688 bytes)
*** Preparing to test memory region 7f32cfbce000 (4096 bytes)
*** Preparing to test memory region 7f32cfbd1000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.1:M 19 Mar 2020 15:50:26.576 # KeyDB 5.3.2 crashed by signal: 4
1:M 19 Mar 2020 15:50:26.576 # Crashed running the instruction at: 0x7f32c6378a9b
1:M 19 Mar 2020 15:50:26.576 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
[0x7f32c6378a9b]

Backtrace:
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x5643fdd2a55b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x5643fdd2acf9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f32ce5ca890]
[0x7f32c6378a9b]

------ INFO OUTPUT ------

To Reproduce
For now not sure how to reproduce it as we hit then when keydb is under load.

We are using https://docs.keydb.dev/docs/commands/#expiremember thought.

[botzill]

Just wandering, if we are using expiremember and we delete some key that is set to expire, can this cause issues?

We have a scenario where we set the ttl and as well can delete the key.

[JohnSully]

The assert is indicating that the expire set is not sorted. I will try to repro by stress testing deleting keys with expiremember set.

[botzill]

We now do not remove the keys with ttl, but still have a crash,

Could this be because of expiremember feature itself?

1:M 19 Mar 2020 19:56:54.858 # KeyDB 5.3.2 crashed by signal: 11
1:M 19 Mar 2020 19:56:54.858 # Crashed running the instruction at: 0x7fae99c258f0
1:M 19 Mar 2020 19:56:54.858 # Accessing address: (nil)
1:M 19 Mar 2020 19:56:54.858 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7fae99c258f0]

Backtrace:
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x55954c39455b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x55954c394cf9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7fae99fe8890]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7fae99c258f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x3039a) [0x7fae99c1539a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30412) [0x7fae99c15412]
keydb-server *:6379(activeExpireCycle(int)+0x513) [0x55954c3cd923]
keydb-server :6379(databasesCron()+0x38) [0x55954c338bd8]
keydb-server :6379(serverCron(aeEventLoop, long long, void)+0x28c) [0x55954c33bb3c]
keydb-server *:6379(aeProcessEvents+0x25b) [0x55954c33364b]
keydb-server *:6379(aeMain+0x45) [0x55954c333925]
keydb-server :6379(workerThreadMain(void)+0x97) [0x55954c337e87]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7fae99fdd6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7fae99d0688f]

------ INFO OUTPUT ------

Server

redis_version:5.3.2
redis_git_sha1:e523afa7
redis_git_dirty:0
redis_build_id:642427f2cdc0fe4e
redis_mode:standalone
os:Linux 4.19.0-0.bpo.6-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.4.0
process_id:1
run_id:e3448703c8d6ce66cd7d11e9cd803b36cb933139
tcp_port:6379
uptime_in_seconds:12737
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7589510
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

Clients

connected_clients:122
client_recent_max_input_buffer:2490
client_recent_max_output_buffer:0
blocked_clients:1
current_client_thread:0
thread_0_clients:50
thread_1_clients:50
thread_2_clients:22
thread_3_clients:0

Memory

used_memory:198267608
used_memory_human:189.08M
used_memory_rss:237973504
used_memory_rss_human:226.95M
used_memory_peak:198349072
used_memory_peak_human:189.16M
used_memory_peak_perc:99.96%
used_memory_overhead:17113578
used_memory_startup:5457504
used_memory_dataset:181154030
used_memory_dataset_perc:93.95%
allocator_allocated:198827576
allocator_active:200642560
allocator_resident:208990208
total_system_memory:33732517888
total_system_memory_human:31.42G
used_memory_lua:82944
used_memory_lua_human:81.00K
used_memory_scripts:17608
used_memory_scripts_human:17.20K
number_of_cached_scripts:9
maxmemory:0
maxmemory_human:0B
maxmemory_policy:allkeys-lfu
allocator_frag_ratio:1.01
allocator_frag_bytes:1814984
allocator_rss_ratio:1.04
allocator_rss_bytes:8347648
rss_overhead_ratio:1.14
rss_overhead_bytes:28983296
mem_fragmentation_ratio:1.20
mem_fragmentation_bytes:39706920
mem_not_counted_for_evict:3462
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:2192868
mem_aof_buffer:3462
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:1235272
rdb_bgsave_in_progress:0
rdb_last_save_time:1584635077
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:4
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:33009664
aof_current_size:119922799
aof_base_size:94200923
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Stats

total_connections_received:272
total_commands_processed:5104038
instantaneous_ops_per_sec:321
total_net_input_bytes:601084251
total_net_output_bytes:332225247
instantaneous_input_kbps:32.12
instantaneous_output_kbps:12.46
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:5513
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:3119031
keyspace_misses:198428
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:11294
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:e78df3d4742e634f85e0e9387d68b1e2c50fb965
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:252.049380
used_cpu_user:164.921152
used_cpu_sys_children:0.868301
used_cpu_user_children:6.517574
server_threads:4
long_lock_waits:8

Commandstats

cmdstat_zcount:calls=1,usec=2,usec_per_call=2.00
cmdstat_set:calls=87738,usec=494522,usec_per_call=5.64
cmdstat_zrange:calls=28089,usec=169673,usec_per_call=6.04
cmdstat_config:calls=2,usec=211,usec_per_call=105.50
cmdstat_lrem:calls=20243,usec=319799,usec_per_call=15.80
cmdstat_expiremember:calls=31,usec=330,usec_per_call=10.65
cmdstat_srem:calls=74789,usec=316223,usec_per_call=4.23
cmdstat_linsert:calls=1,usec=2233,usec_per_call=2233.00
cmdstat_subscribe:calls=8,usec=43,usec_per_call=5.38
cmdstat_del:calls=7890,usec=29012,usec_per_call=3.68
cmdstat_exec:calls=5488,usec=816646,usec_per_call=148.81
cmdstat_hscan:calls=1,usec=38,usec_per_call=38.00
cmdstat_incr:calls=152765,usec=801342,usec_per_call=5.25
cmdstat_llen:calls=16,usec=26,usec_per_call=1.62
cmdstat_client:calls=5,usec=15,usec_per_call=3.00
cmdstat_rpoplpush:calls=12810,usec=147835,usec_per_call=11.54
cmdstat_lpush:calls=157821,usec=1055725,usec_per_call=6.69
cmdstat_multi:calls=5488,usec=6570,usec_per_call=1.20
cmdstat_hset:calls=187916,usec=1333306,usec_per_call=7.10
cmdstat_hincrby:calls=1,usec=5,usec_per_call=5.00
cmdstat_eval:calls=103,usec=4049,usec_per_call=39.31
cmdstat_ping:calls=1,usec=2,usec_per_call=2.00
cmdstat_zcard:calls=15,usec=10,usec_per_call=0.67
cmdstat_hexists:calls=308162,usec=2916789,usec_per_call=9.47
cmdstat_zrem:calls=17958,usec=74100,usec_per_call=4.13
cmdstat_brpoplpush:calls=787,usec=8673,usec_per_call=11.02
cmdstat_get:calls=92906,usec=656601,usec_per_call=7.07
cmdstat_hget:calls=27313,usec=274816,usec_per_call=10.06
cmdstat_hincrbyfloat:calls=381901,usec=8270794,usec_per_call=21.66
cmdstat_evalsha:calls=274695,usec=30825665,usec_per_call=112.22
cmdstat_hmget:calls=2202470,usec=7555661,usec_per_call=3.43
cmdstat_zrangebyscore:calls=28080,usec=179498,usec_per_call=6.39
cmdstat_script:calls=5377,usec=26545,usec_per_call=4.94
cmdstat_exists:calls=174118,usec=250356,usec_per_call=1.44
cmdstat_auth:calls=275,usec=1180,usec_per_call=4.29
cmdstat_zrevrange:calls=83,usec=285,usec_per_call=3.43
cmdstat_object:calls=1,usec=4,usec_per_call=4.00
cmdstat_lrange:calls=402,usec=7461,usec_per_call=18.56
cmdstat_type:calls=101,usec=59,usec_per_call=0.58
cmdstat_scan:calls=1,usec=177,usec_per_call=177.00
cmdstat_rpush:calls=1,usec=5,usec_per_call=5.00
cmdstat_info:calls=1590,usec=223778,usec_per_call=140.74
cmdstat_zadd:calls=12816,usec=236059,usec_per_call=18.42
cmdstat_smembers:calls=304,usec=4504,usec_per_call=14.82
cmdstat_hgetall:calls=455268,usec=5785947,usec_per_call=12.71
cmdstat_zscore:calls=126,usec=552,usec_per_call=4.38
cmdstat_hlen:calls=2,usec=44,usec_per_call=22.00
cmdstat_sadd:calls=279,usec=9133,usec_per_call=32.73
cmdstat_hdel:calls=451,usec=1610,usec_per_call=3.57
cmdstat_pttl:calls=1,usec=7,usec_per_call=7.00
cmdstat_publish:calls=211612,usec=389627,usec_per_call=1.84
cmdstat_lindex:calls=1,usec=1,usec_per_call=1.00
cmdstat_hmset:calls=165734,usec=2268251,usec_per_call=13.69
cmdstat_keys:calls=1,usec=64,usec_per_call=64.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=152909,expires=109,avg_ttl=14546

------ CLIENT LIST OUTPUT ------
id=270 addr=10.244.5.224:57656 fd=180 name= age=1446 idle=1446 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=272 addr=10.244.5.224:57660 fd=208 name= age=1446 idle=1446 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=179 addr=10.244.5.224:36190 fd=138 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=186 addr=10.244.5.224:35156 fd=141 name= age=11094 idle=11094 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=190 addr=10.244.5.224:35218 fd=144 name= age=11092 idle=11092 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=180 addr=10.244.5.224:36196 fd=139 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=214 addr=10.244.5.224:54776 fd=163 name= age=8248 idle=8248 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=182 addr=10.244.5.224:36198 fd=140 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=139 addr=10.244.5.224:34954 fd=108 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=251 addr=10.244.5.224:60156 fd=193 name= age=2906 idle=2906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=253 addr=10.244.5.224:60160 fd=195 name= age=2906 idle=2906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=262 addr=10.244.5.224:60278 fd=202 name= age=2900 idle=2899 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=254 addr=10.244.5.224:60162 fd=194 name= age=2906 idle=2906 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=141 addr=10.244.5.224:34958 fd=110 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=162 addr=10.244.5.224:35398 fd=124 name= age=12378 idle=12378 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=142 addr=10.244.5.224:34960 fd=109 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=213 addr=10.244.5.224:54774 fd=164 name= age=8248 idle=8248 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=127 addr=10.244.5.224:33482 fd=97 name= age=12524 idle=12524 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=195 addr=10.244.5.224:54228 fd=150 name= age=10106 idle=10106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=197 addr=10.244.5.224:54232 fd=152 name= age=10106 idle=10106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=221 addr=10.244.5.224:41218 fd=170 name= age=7301 idle=7301 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=274 addr=10.244.5.224:59768 fd=210 name= age=1332 idle=1332 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=222 addr=10.244.5.224:41220 fd=169 name= age=7301 idle=7301 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=198 addr=10.244.5.224:54234 fd=151 name= age=10106 idle=10106 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=187 addr=10.244.5.224:35212 fd=143 name= age=11092 idle=11092 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=189 addr=10.244.5.224:35216 fd=145 name= age=11092 idle=11092 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=211 addr=10.244.5.224:54770 fd=162 name= age=8248 idle=8248 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=203 addr=10.244.5.224:53002 fd=155 name= age=8373 idle=8373 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=205 addr=10.244.5.224:53006 fd=157 name= age=8373 idle=8373 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=219 addr=10.244.5.224:41212 fd=168 name= age=7301 idle=7301 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=206 addr=10.244.5.224:53008 fd=156 name= age=8373 idle=8373 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=145 addr=10.244.5.224:34966 fd=113 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=185 addr=10.244.5.224:35150 fd=142 name= age=11094 idle=11094 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=172 addr=10.244.5.224:36170 fd=132 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=215 addr=10.244.5.224:33200 fd=165 name= age=7825 idle=7825 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=217 addr=10.244.5.224:33204 fd=167 name= age=7825 idle=7825 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=218 addr=10.244.5.224:33206 fd=166 name= age=7825 idle=7825 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=175 addr=10.244.5.224:36180 fd=133 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=174 addr=10.244.5.224:36174 fd=134 name= age=12320 idle=12320 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=146 addr=10.244.5.224:34968 fd=112 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=183 addr=10.244.5.224:35146 fd=83 name= age=11094 idle=11094 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=138 addr=10.244.5.224:33802 fd=106 name= age=12502 idle=12502 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=123 addr=10.244.5.224:33398 fd=94 name= age=12529 idle=12529 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=124 addr=10.244.5.224:33464 fd=96 name= age=12525 idle=12525 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=135 addr=10.244.5.224:33794 fd=105 name= age=12502 idle=12502 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=158 addr=10.244.5.224:35386 fd=122 name= age=12379 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=163 addr=10.244.5.224:36092 fd=126 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=165 addr=10.244.5.224:36096 fd=128 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=166 addr=10.244.5.224:36098 fd=127 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=169 addr=10.244.5.224:36106 fd=131 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=266 addr=10.244.5.224:32814 fd=205 name= age=2854 idle=2854 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=268 addr=10.244.5.224:32818 fd=207 name= age=2854 idle=2854 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=269 addr=10.244.5.224:32820 fd=206 name= age=2854 idle=2854 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=170 addr=10.244.5.224:36108 fd=130 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=147 addr=10.244.5.224:34974 fd=100 name= age=12410 idle=12410 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=148 addr=10.244.5.224:35312 fd=114 name= age=12384 idle=12384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=150 addr=10.244.5.224:35316 fd=116 name= age=12384 idle=12384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=167 addr=10.244.5.224:36102 fd=129 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=151 addr=10.244.5.224:35318 fd=115 name= age=12384 idle=12384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=245 addr=10.244.5.224:43336 fd=189 name= age=3831 idle=3831 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=246 addr=10.244.5.224:43340 fd=188 name= age=3831 idle=3831 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=161 addr=10.244.5.224:35396 fd=125 name= age=12378 idle=12378 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=238 addr=10.244.5.224:48394 fd=182 name= age=4981 idle=4981 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=155 addr=10.244.5.224:35380 fd=118 name= age=12379 idle=12379 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=226 addr=10.244.5.224:37260 fd=171 name= age=5703 idle=5703 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=143 addr=10.244.5.224:34962 fd=111 name= age=12411 idle=12411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=137 addr=10.244.5.224:33800 fd=107 name= age=12502 idle=12502 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=159 addr=10.244.5.224:35392 fd=123 name= age=12378 idle=12378 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=126 addr=10.244.5.224:33468 fd=98 name= age=12525 idle=12525 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=235 addr=10.244.5.224:48388 fd=181 name= age=4981 idle=4981 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=237 addr=10.244.5.224:48392 fd=183 name= age=4981 idle=4981 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=234 addr=10.244.5.224:37814 fd=178 name= age=5668 idle=5668 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=128 addr=10.244.5.224:33484 fd=99 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=12524 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=223 addr=10.244.5.224:37254 fd=161 name= age=5703 idle=5703 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=225 addr=10.244.5.224:37258 fd=172 name= age=5703 idle=5703 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=156 addr=10.244.5.224:35382 fd=120 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=12379 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=130 addr=10.244.5.224:33488 fd=101 name= age=12524 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=230 addr=10.244.5.224:37740 fd=175 name= age=5673 idle=5673 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=242 addr=10.244.5.224:56402 fd=185 name= age=4587 idle=4587 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=259 addr=10.244.5.224:60270 fd=199 name=queue:ZWNhZWEwOWYtYjYyZC00NzZkLTljNTQtMTFlYmE1ODRhMmZm age=2900 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=260 addr=10.244.5.224:60272 fd=200 name= age=2900 idle=0 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=2484 qbuf-free=34976 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=261 addr=10.244.5.224:60274 fd=201 name= age=2900 idle=2 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=231 addr=10.244.5.224:37808 fd=177 name= age=5668 idle=5668 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=233 addr=10.244.5.224:37812 fd=179 name= age=5668 idle=5668 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=90 addr=10.244.5.184:45978 fd=174 name= age=12645 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=207 addr=10.244.5.224:53272 fd=158 name= age=8353 idle=8353 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=209 addr=10.244.5.224:53278 fd=160 name= age=8353 idle=8353 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=250 addr=10.244.5.224:59224 fd=191 name= age=2962 idle=2962 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=210 addr=10.244.5.224:53280 fd=159 name= age=8353 idle=8353 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=239 addr=10.244.5.224:56396 fd=184 name= age=4587 idle=4587 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=241 addr=10.244.5.224:56400 fd=186 name= age=4587 idle=4587 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=202 addr=10.244.5.224:52684 fd=153 name= age=8395 idle=8395 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=227 addr=10.244.5.224:37734 fd=173 name= age=5673 idle=5673 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=229 addr=10.244.5.224:37738 fd=176 name= age=5673 idle=5673 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=194 addr=10.244.5.224:35288 fd=147 name= age=11088 idle=11088 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=119 addr=10.244.5.224:32956 fd=92 name= age=12562 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hget
id=120 addr=10.244.5.224:33392 fd=93 name= age=12529 idle=12529 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=176 addr=10.244.5.224:36182 fd=135 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=12320 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=177 addr=10.244.5.224:36184 fd=136 name= age=12320 idle=12285 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=243 addr=10.244.5.224:43332 fd=187 name= age=3831 idle=3831 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=201 addr=10.244.5.224:52682 fd=154 name= age=8395 idle=8395 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=178 addr=10.244.5.224:36186 fd=137 name= age=12320 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=171 addr=10.244.5.224:36112 fd=121 name= age=12325 idle=12325 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=152 addr=10.244.5.224:35374 fd=117 name= age=12379 idle=12379 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=191 addr=10.244.5.224:35282 fd=146 name= age=11088 idle=11088 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=247 addr=10.244.5.224:59218 fd=190 name= age=2962 idle=2962 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=249 addr=10.244.5.224:59222 fd=192 name= age=2962 idle=2962 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=265 addr=10.244.5.224:60284 fd=203 name= age=2899 idle=2899 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=193 addr=10.244.5.224:35286 fd=148 name= age=11088 idle=11088 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=199 addr=10.244.5.224:52678 fd=149 name= age=8395 idle=8395 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=154 addr=10.244.5.224:35378 fd=119 name= age=12379 idle=12379 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=273 addr=10.244.5.224:57662 fd=209 name= age=1446 idle=1446 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=122 addr=10.244.5.224:33396 fd=95 name= age=12529 idle=12529 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=131 addr=10.244.5.224:33494 fd=102 name= age=12524 idle=12524 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=255 addr=10.244.5.224:60256 fd=196 name= age=2900 idle=2900 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=257 addr=10.244.5.224:60260 fd=198 name= age=2900 idle=2900 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=258 addr=10.244.5.224:60262 fd=197 name= age=2900 idle=2900 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=133 addr=10.244.5.224:33498 fd=104 name= age=12524 idle=12524 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=276 addr=10.244.5.224:59772 fd=212 name= age=1332 idle=1332 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=277 addr=10.244.5.224:59774 fd=211 name= age=1332 idle=1332 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=134 addr=10.244.5.224:33500 fd=103 name= age=12524 idle=12524 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=264 addr=10.244.5.224:60282 fd=204 name= age=2900 idle=2900 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info

------ REGISTERS ------
1:M 19 Mar 2020 19:56:54.860 #
RAX:0000000000000004 RBX:0000000000000000
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000002 RSI:00007fae91e7a0f0
RBP:00007fae99d9c7d8 RSP:00007fae91e7a210
R8 :0000000000000000 R9 :00007fae91e7a0f0
R10:0000000000000008 R11:0000000000000246
R12:000055954c48757f R13:000055954c49be78
R14:0000000000000144 R15:8000000000000000
RIP:00007fae99c258f0 EFL:0000000000010246
CSGSFS:002b000000000033
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21f) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21e) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21d) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21c) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21b) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a21a) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a219) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a218) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a217) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a216) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a215) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a214) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a213) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.860 # (00007fae91e7a212) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.861 # (00007fae91e7a211) -> 0000000000000000
1:M 19 Mar 2020 19:56:54.861 # (00007fae91e7a210) -> 0000000000000020

------ FAST MEMORY TEST ------
1:M 19 Mar 2020 19:56:54.861 # Bio thread for job type #0 terminated
1:M 19 Mar 2020 19:56:54.861 # Bio thread for job type #1 terminated
1:M 19 Mar 2020 19:56:54.861 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55954c6e3000 (2260992 bytes)
*** Preparing to test memory region 55954c91d000 (323584 bytes)
*** Preparing to test memory region 7fae74a00000 (190976000 bytes)
*** Preparing to test memory region 7fae84000000 (135168 bytes)
*** Preparing to test memory region 7fae88000000 (348160 bytes)
*** Preparing to test memory region 7fae8c080000 (58195968 bytes)
*** Preparing to test memory region 7fae8f9ff000 (8388608 bytes)
*** Preparing to test memory region 7fae901ff000 (2097152 bytes)
*** Preparing to test memory region 7fae90400000 (8388608 bytes)
*** Preparing to test memory region 7fae90c00000 (2097152 bytes)
*** Preparing to test memory region 7fae90e7c000 (8388608 bytes)
*** Preparing to test memory region 7fae9167d000 (8388608 bytes)
*** Preparing to test memory region 7fae91e7e000 (8388608 bytes)
*** Preparing to test memory region 7fae9267f000 (8388608 bytes)
*** Preparing to test memory region 7fae92e80000 (8388608 bytes)
*** Preparing to test memory region 7fae93680000 (14155776 bytes)
*** Preparing to test memory region 7fae94771000 (188416 bytes)
*** Preparing to test memory region 7fae94aa7000 (4096 bytes)
*** Preparing to test memory region 7fae94cf1000 (4096 bytes)
*** Preparing to test memory region 7fae9577d000 (4096 bytes)
*** Preparing to test memory region 7fae95cac000 (4096 bytes)
*** Preparing to test memory region 7fae965fb000 (4096 bytes)
*** Preparing to test memory region 7fae96c71000 (8192 bytes)
*** Preparing to test memory region 7fae972b3000 (4096 bytes)
*** Preparing to test memory region 7fae97fda000 (4096 bytes)
*** Preparing to test memory region 7fae989d4000 (8192 bytes)
*** Preparing to test memory region 7fae990e9000 (12288 bytes)
*** Preparing to test memory region 7fae99fd2000 (16384 bytes)
*** Preparing to test memory region 7fae9a1f1000 (16384 bytes)
*** Preparing to test memory region 7fae9ab30000 (16384 bytes)
*** Preparing to test memory region 7fae9b5ce000 (114688 bytes)
*** Preparing to test memory region 7fae9b5ec000 (4096 bytes)
*** Preparing to test memory region 7fae9b5ef000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.%

[botzill]

It was running all night and it constantly is crashing after some time, we had like 5-6 crashes this night.

1:M 20 Mar 2020 06:52:47.888 # KeyDB 5.3.2 crashed by signal: 11
1:M 20 Mar 2020 06:52:47.888 # Crashed running the instruction at: 0x7f5bda9058f0
1:M 20 Mar 2020 06:52:47.888 # Accessing address: (nil)
1:M 20 Mar 2020 06:52:47.888 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7f5bda9058f0]

Backtrace:
keydb-server :6379(logStackTrace(ucontext_t)+0x6b) [0x55c8313ad55b]
keydb-server :6379(sigsegvHandler(int, siginfo_t, void*)+0xc9) [0x55c8313adcf9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f5bdacc8890]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x230) [0x7f5bda9058f0]
/lib/x86_64-linux-gnu/libc.so.6(+0x3039a) [0x7f5bda8f539a]
/lib/x86_64-linux-gnu/libc.so.6(+0x30412) [0x7f5bda8f5412]
keydb-server *:6379(activeExpireCycle(int)+0x513) [0x55c8313e6923]
keydb-server :6379(databasesCron()+0x38) [0x55c831351bd8]
keydb-server :6379(serverCron(aeEventLoop, long long, void)+0x28c) [0x55c831354b3c]
keydb-server *:6379(aeProcessEvents+0x25b) [0x55c83134c64b]
keydb-server *:6379(aeMain+0x45) [0x55c83134c925]
keydb-server :6379(workerThreadMain(void)+0x97) [0x55c831350e87]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f5bdacbd6db]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f5bda9e688f]

------ INFO OUTPUT ------

Server

redis_version:5.3.2
redis_git_sha1:e523afa7
redis_git_dirty:0
redis_build_id:642427f2cdc0fe4e
redis_mode:standalone
os:Linux 4.19.0-0.bpo.6-amd64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:7.4.0
process_id:1
run_id:b3ba49e8f50949181e05389657a8a632b3c4224f
tcp_port:6379
uptime_in_seconds:25145
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7628863
executable:/data/keydb-server
config_file:/etc/keydb/keydb.conf

Clients

connected_clients:145
client_recent_max_input_buffer:2628
client_recent_max_output_buffer:0
blocked_clients:1
current_client_thread:0
thread_0_clients:51
thread_1_clients:50
thread_2_clients:44
thread_3_clients:0

Memory

used_memory:359829320
used_memory_human:343.16M
used_memory_rss:450949120
used_memory_rss_human:430.06M
used_memory_peak:359865144
used_memory_peak_human:343.19M
used_memory_peak_perc:99.99%
used_memory_overhead:25393720
used_memory_startup:5457504
used_memory_dataset:334435600
used_memory_dataset_perc:94.37%
allocator_allocated:360587792
allocator_active:365568000
allocator_resident:376459264
total_system_memory:33732517888
total_system_memory_human:31.42G
used_memory_lua:95232
used_memory_lua_human:93.00K
used_memory_scripts:16712
used_memory_scripts_human:16.32K
number_of_cached_scripts:9
maxmemory:0
maxmemory_human:0B
maxmemory_policy:allkeys-lfu
allocator_frag_ratio:1.01
allocator_frag_bytes:4980208
allocator_rss_ratio:1.03
allocator_rss_bytes:10891264
rss_overhead_ratio:1.20
rss_overhead_bytes:74489856
mem_fragmentation_ratio:1.25
mem_fragmentation_bytes:91120824
mem_not_counted_for_evict:1620
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:2552020
mem_aof_buffer:1620
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:1961546
rdb_bgsave_in_progress:0
rdb_last_save_time:1584662022
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
aof_current_size:394122703
aof_base_size:316759535
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

Stats

total_connections_received:354
total_commands_processed:18147668
instantaneous_ops_per_sec:289
total_net_input_bytes:1850052040
total_net_output_bytes:712013876
instantaneous_input_kbps:30.21
instantaneous_output_kbps:15.07
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:14953
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:16474235
keyspace_misses:25199
pubsub_channels:2
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:9b1035c43303b15d89dfc7a04ae64e93960b1e76
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:575.021292
used_cpu_user:463.181004
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:2062

Commandstats

cmdstat_keys:calls=1,usec=57335,usec_per_call=57335.00
cmdstat_lpush:calls=19266,usec=129178,usec_per_call=6.70
cmdstat_expiremember:calls=33,usec=443,usec_per_call=13.42
cmdstat_zrevrange:calls=20,usec=369,usec_per_call=18.45
cmdstat_rpoplpush:calls=13966,usec=184695,usec_per_call=13.22
cmdstat_publish:calls=104858,usec=1030890,usec_per_call=9.83
cmdstat_smembers:calls=1008,usec=12626,usec_per_call=12.53
cmdstat_zrem:calls=25135,usec=114669,usec_per_call=4.56
cmdstat_zrangebyscore:calls=57253,usec=552946,usec_per_call=9.66
cmdstat_get:calls=181377,usec=1122326,usec_per_call=6.19
cmdstat_rename:calls=2,usec=11,usec_per_call=5.50
cmdstat_script:calls=13845,usec=80121,usec_per_call=5.79
cmdstat_brpoplpush:calls=3991,usec=53780,usec_per_call=13.48
cmdstat_auth:calls=354,usec=1687,usec_per_call=4.77
cmdstat_srem:calls=153672,usec=541150,usec_per_call=3.52
cmdstat_hincrbyfloat:calls=18765,usec=461393,usec_per_call=24.59
cmdstat_hmget:calls=16082627,usec=44497339,usec_per_call=2.77
cmdstat_hincrby:calls=100,usec=271,usec_per_call=2.71
cmdstat_hget:calls=14789,usec=93778,usec_per_call=6.34
cmdstat_zscore:calls=3639,usec=6980,usec_per_call=1.92
cmdstat_zcard:calls=3,usec=7,usec_per_call=2.33
cmdstat_subscribe:calls=7,usec=49,usec_per_call=7.00
cmdstat_client:calls=3,usec=13,usec_per_call=4.33
cmdstat_sadd:calls=508,usec=28227,usec_per_call=55.56
cmdstat_hexists:calls=13995,usec=155761,usec_per_call=11.13
cmdstat_del:calls=715,usec=2508,usec_per_call=3.51
cmdstat_eval:calls=182,usec=13041,usec_per_call=71.65
cmdstat_rpush:calls=100,usec=379,usec_per_call=3.79
cmdstat_incr:calls=5411,usec=34576,usec_per_call=6.39
cmdstat_hmset:calls=19679,usec=307726,usec_per_call=15.64
cmdstat_info:calls=441,usec=59723,usec_per_call=135.43
cmdstat_evalsha:calls=247639,usec=144028650,usec_per_call=581.61
cmdstat_hset:calls=45277,usec=352015,usec_per_call=7.77
cmdstat_lrem:calls=31546,usec=822835,usec_per_call=26.08
cmdstat_set:calls=168643,usec=908663,usec_per_call=5.39
cmdstat_hgetall:calls=48380,usec=778617,usec_per_call=16.09
cmdstat_exists:calls=36234,usec=82347,usec_per_call=2.27
cmdstat_exec:calls=13878,usec=3390332,usec_per_call=244.30
cmdstat_lrange:calls=2853,usec=35373759,usec_per_call=12398.79
cmdstat_zadd:calls=14069,usec=150944,usec_per_call=10.73
cmdstat_zrange:calls=57253,usec=534467,usec_per_call=9.34
cmdstat_multi:calls=13878,usec=17269,usec_per_call=1.24
cmdstat_llen:calls=3,usec=6,usec_per_call=2.00
cmdstat_hdel:calls=80,usec=446,usec_per_call=5.57

Cluster

cluster_enabled:0

Keyspace

db0:keys=274296,expires=109,avg_ttl=16048

------ CLIENT LIST OUTPUT ------
id=292 addr=10.244.5.98:60282 fd=138 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=294 addr=10.244.5.98:60270 fd=132 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=296 addr=10.244.5.98:60308 fd=139 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=318 addr=10.244.5.98:60772 fd=113 name= age=983 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=319 addr=10.244.5.98:60796 fd=121 name= age=983 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=298 addr=10.244.5.98:60310 fd=141 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=300 addr=10.244.5.98:60278 fd=140 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=302 addr=10.244.5.98:60272 fd=127 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=266 addr=10.244.5.98:60402 fd=181 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=267 addr=10.244.5.98:60420 fd=182 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=353 addr=10.244.5.98:33106 fd=231 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=949 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=354 addr=10.244.5.98:33110 fd=232 name= age=949 idle=948 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=355 addr=10.244.5.98:33112 fd=233 name= age=949 idle=2 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=268 addr=10.244.5.98:60426 fd=183 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=269 addr=10.244.5.98:60434 fd=184 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=169 addr=10.244.5.98:58152 fd=92 name= age=1164 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=170 addr=10.244.5.98:58158 fd=93 name=queue:ZWNhZWEwOWYtYjYyZC00NzZkLTljNTQtMTFlYmE1ODRhMmZm age=1164 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=184 addr=10.244.5.98:59506 fd=105 name= age=1067 idle=1067 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=186 addr=10.244.5.98:59510 fd=107 name= age=1067 idle=1067 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=187 addr=10.244.5.98:59512 fd=106 name= age=1067 idle=1067 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=313 addr=10.244.5.98:60818 fd=213 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=233 addr=10.244.5.98:60376 fd=150 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=234 addr=10.244.5.98:60440 fd=202 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=235 addr=10.244.5.98:60390 fd=151 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=236 addr=10.244.5.98:60442 fd=203 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=237 addr=10.244.5.98:60404 fd=152 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=228 addr=10.244.5.98:60418 fd=199 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=230 addr=10.244.5.98:60428 fd=200 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=317 addr=10.244.5.98:60824 fd=147 name= age=983 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=232 addr=10.244.5.98:60432 fd=201 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=321 addr=10.244.5.98:32808 fd=120 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=322 addr=10.244.5.98:32814 fd=148 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=323 addr=10.244.5.98:32810 fd=149 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=324 addr=10.244.5.98:32812 fd=167 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=325 addr=10.244.5.98:32816 fd=170 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=326 addr=10.244.5.98:32818 fd=171 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=327 addr=10.244.5.98:32822 fd=172 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=328 addr=10.244.5.98:32820 fd=169 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=329 addr=10.244.5.98:32824 fd=173 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=330 addr=10.244.5.98:32826 fd=174 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=331 addr=10.244.5.98:32828 fd=175 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=332 addr=10.244.5.98:32830 fd=190 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=333 addr=10.244.5.98:32832 fd=176 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=334 addr=10.244.5.98:32834 fd=191 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=335 addr=10.244.5.98:32836 fd=192 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=351 addr=10.244.5.98:33082 fd=226 name= age=949 idle=949 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=352 addr=10.244.5.98:33084 fd=230 name= age=949 idle=949 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=336 addr=10.244.5.98:32838 fd=193 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=337 addr=10.244.5.98:32840 fd=194 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=338 addr=10.244.5.98:32842 fd=214 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=339 addr=10.244.5.98:32848 fd=215 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=340 addr=10.244.5.98:32844 fd=216 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=341 addr=10.244.5.98:32846 fd=217 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=342 addr=10.244.5.98:32854 fd=218 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=343 addr=10.244.5.98:32850 fd=219 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=344 addr=10.244.5.98:32852 fd=220 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=312 addr=10.244.5.98:60808 fd=205 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=222 addr=10.244.5.98:60398 fd=196 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=350 addr=10.244.5.98:32944 fd=228 name= age=949 idle=949 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=223 addr=10.244.5.98:60286 fd=143 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=224 addr=10.244.5.98:60406 fd=197 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=226 addr=10.244.5.98:60416 fd=198 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=188 addr=10.244.5.98:59516 fd=108 name= age=1066 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=220 addr=10.244.5.98:60388 fd=195 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=309 addr=10.244.5.98:60774 fd=142 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=310 addr=10.244.5.98:60798 fd=164 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=311 addr=10.244.5.98:60802 fd=165 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=190 addr=10.244.5.98:59522 fd=110 name= age=1066 idle=1066 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=320 addr=10.244.5.98:60834 fd=166 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=179 addr=10.244.5.98:59124 fd=100 name= age=1097 idle=1097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=171 addr=10.244.5.98:58160 fd=94 name= age=1164 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=2622 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=brpoplpush
id=172 addr=10.244.5.98:58162 fd=95 name= age=1164 idle=5 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=176 addr=10.244.5.98:59116 fd=99 name= age=1097 idle=1097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=183 addr=10.244.5.98:59454 fd=103 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=358 addr=10.244.5.98:33124 fd=236 name= age=948 idle=948 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=359 addr=10.244.5.98:33126 fd=235 name= age=948 idle=948 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=178 addr=10.244.5.98:59120 fd=101 name= age=1097 idle=1097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=173 addr=10.244.5.98:58164 fd=96 name=queue:OThmNGFjYzMtMmJmOS00Yzc3LWFjZDYtMGFkNzE3MzQ4ZGI0 age=1164 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exec
id=180 addr=10.244.5.98:59448 fd=102 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=182 addr=10.244.5.98:59452 fd=104 name= age=1070 idle=1070 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=175 addr=10.244.5.98:58168 fd=98 name= age=1164 idle=2 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=191 addr=10.244.5.98:59754 fd=117 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=192 addr=10.244.5.98:59758 fd=118 name= age=1046 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=193 addr=10.244.5.98:59764 fd=119 name= age=1046 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=211 addr=10.244.5.98:60296 fd=187 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=212 addr=10.244.5.98:60300 fd=188 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=348 addr=10.244.5.98:32940 fd=229 name= age=949 idle=949 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=213 addr=10.244.5.98:60306 fd=189 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=196 addr=10.244.5.98:59772 fd=122 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=356 addr=10.244.5.98:33120 fd=234 name= age=948 idle=948 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=197 addr=10.244.5.98:59756 fd=97 name= age=1046 idle=1027 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=199 addr=10.244.5.98:59760 fd=112 name= age=1046 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=201 addr=10.244.5.98:59774 fd=114 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=202 addr=10.244.5.98:59776 fd=115 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=209 addr=10.244.5.98:60288 fd=185 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=210 addr=10.244.5.98:60292 fd=186 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=203 addr=10.244.5.98:59778 fd=116 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=206 addr=10.244.5.98:59980 fd=124 name= age=1027 idle=1027 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=207 addr=10.244.5.98:59984 fd=125 name= age=1027 idle=1027 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=204 addr=10.244.5.98:59804 fd=109 name= age=1046 idle=1046 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=208 addr=10.244.5.98:60280 fd=168 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=316 addr=10.244.5.98:60832 fd=145 name= age=983 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=238 addr=10.244.5.98:60408 fd=153 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=239 addr=10.244.5.98:60290 fd=146 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=240 addr=10.244.5.98:60410 fd=154 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=241 addr=10.244.5.98:60412 fd=156 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=242 addr=10.244.5.98:60414 fd=157 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=243 addr=10.244.5.98:60422 fd=158 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=244 addr=10.244.5.98:60424 fd=159 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=245 addr=10.244.5.98:60294 fd=155 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=246 addr=10.244.5.98:60430 fd=160 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=247 addr=10.244.5.98:60436 fd=162 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=248 addr=10.244.5.98:60298 fd=161 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=315 addr=10.244.5.98:60830 fd=144 name= age=983 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=249 addr=10.244.5.98:60304 fd=163 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=262 addr=10.244.5.98:60380 fd=177 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=263 addr=10.244.5.98:60384 fd=178 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=264 addr=10.244.5.98:60394 fd=179 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=265 addr=10.244.5.98:60396 fd=180 name= age=1022 idle=1022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=270 addr=10.244.5.98:60058 fd=111 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=271 addr=10.244.5.98:60244 fd=123 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=272 addr=10.244.5.98:60258 fd=126 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=273 addr=10.244.5.98:60056 fd=204 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=274 addr=10.244.5.98:60248 fd=128 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=276 addr=10.244.5.98:60250 fd=129 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=277 addr=10.244.5.98:60254 fd=206 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=278 addr=10.244.5.98:60266 fd=130 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=279 addr=10.244.5.98:60264 fd=207 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=280 addr=10.244.5.98:60252 fd=131 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=281 addr=10.244.5.98:60276 fd=208 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=282 addr=10.244.5.98:60256 fd=133 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=283 addr=10.244.5.98:60284 fd=209 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=284 addr=10.244.5.98:60260 fd=134 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=285 addr=10.244.5.98:60302 fd=210 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=286 addr=10.244.5.98:60262 fd=135 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=288 addr=10.244.5.98:60268 fd=136 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=289 addr=10.244.5.98:60312 fd=212 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=290 addr=10.244.5.98:60274 fd=137 name= age=1022 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=305 addr=10.244.5.98:60382 fd=221 name= age=986 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=314 addr=10.244.5.98:60812 fd=211 name= age=986 idle=983 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=306 addr=10.244.5.98:60386 fd=222 name= age=986 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=307 addr=10.244.5.98:60392 fd=223 name= age=986 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=345 addr=10.244.5.98:32858 fd=225 name= age=967 idle=949 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=evalsha
id=347 addr=10.244.5.98:32862 fd=227 name= age=967 idle=967 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
id=308 addr=10.244.5.98:60400 fd=224 name= age=986 idle=986 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info

------ REGISTERS ------
1:M 20 Mar 2020 06:52:47.890 #
RAX:0000000000000004 RBX:0000000000000000
RCX:0000000000000000 RDX:0000000000000000
RDI:0000000000000002 RSI:00007f5bbd7fd0f0
RBP:00007f5bdaa7c7d8 RSP:00007f5bbd7fd210
R8 :0000000000000000 R9 :00007f5bbd7fd0f0
R10:0000000000000008 R11:0000000000000246
R12:000055c8314a057f R13:000055c8314b4e78
R14:0000000000000144 R15:8000000000000000
RIP:00007f5bda9058f0 EFL:0000000000010246
CSGSFS:002b000000000033
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21f) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21e) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21d) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21c) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21b) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd21a) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd219) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd218) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd217) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd216) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd215) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd214) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd213) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd212) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd211) -> 0000000000000000
1:M 20 Mar 2020 06:52:47.890 # (00007f5bbd7fd210) -> 0000000000000020

------ FAST MEMORY TEST ------
1:M 20 Mar 2020 06:52:47.890 # Bio thread for job type #0 terminated
1:M 20 Mar 2020 06:52:47.890 # Bio thread for job type #1 terminated
1:M 20 Mar 2020 06:52:47.890 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55c8316fc000 (2260992 bytes)
*** Preparing to test memory region 55c83387f000 (323584 bytes)
*** Preparing to test memory region 7f5ba4000000 (37580800 bytes)
*** Preparing to test memory region 7f5ba817b000 (39321600 bytes)
*** Preparing to test memory region 7f5baa7fd000 (12582912 bytes)
*** Preparing to test memory region 7f5bab400000 (12718080 bytes)
*** Preparing to test memory region 7f5bb0000000 (135168 bytes)
*** Preparing to test memory region 7f5bb4000000 (34836480 bytes)
*** Preparing to test memory region 7f5bb8200000 (48234496 bytes)
*** Preparing to test memory region 7f5bbb1ff000 (2097152 bytes)
*** Preparing to test memory region 7f5bbb400000 (8388608 bytes)
*** Preparing to test memory region 7f5bbbc00000 (2097152 bytes)
*** Preparing to test memory region 7f5bbbffe000 (8388608 bytes)
*** Preparing to test memory region 7f5bbc7ff000 (8388608 bytes)
*** Preparing to test memory region 7f5bbd000000 (8388608 bytes)
*** Preparing to test memory region 7f5bbd800000 (65011712 bytes)
*** Preparing to test memory region 7f5bc16fd000 (290979840 bytes)
*** Preparing to test memory region 7f5bd2c7e000 (8388608 bytes)
*** Preparing to test memory region 7f5bd347f000 (8388608 bytes)
*** Preparing to test memory region 7f5bd3c80000 (8388608 bytes)
*** Preparing to test memory region 7f5bd4480000 (14155776 bytes)
*** Preparing to test memory region 7f5bd5451000 (188416 bytes)
*** Preparing to test memory region 7f5bd5787000 (4096 bytes)
*** Preparing to test memory region 7f5bd59d1000 (4096 bytes)
*** Preparing to test memory region 7f5bd645d000 (4096 bytes)
*** Preparing to test memory region 7f5bd698c000 (4096 bytes)
*** Preparing to test memory region 7f5bd72db000 (4096 bytes)
*** Preparing to test memory region 7f5bd7951000 (8192 bytes)
*** Preparing to test memory region 7f5bd7f93000 (4096 bytes)
*** Preparing to test memory region 7f5bd8cba000 (4096 bytes)
*** Preparing to test memory region 7f5bd96b4000 (8192 bytes)
*** Preparing to test memory region 7f5bd9dc9000 (12288 bytes)
*** Preparing to test memory region 7f5bdacb2000 (16384 bytes)
*** Preparing to test memory region 7f5bdaed1000 (16384 bytes)
*** Preparing to test memory region 7f5bdb810000 (16384 bytes)
*** Preparing to test memory region 7f5bdc2ae000 (114688 bytes)
*** Preparing to test memory region 7f5bdc2cc000 (4096 bytes)
*** Preparing to test memory region 7f5bdc2cf000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.%

[botzill]

We did remove the expire keys and seems to work ok, no crashes so far.

[botzill]

@JohnSully do you have any updates it this or when this can be solved? This is crucial for us and want to know if this will be addressed soon?

Thx a lot!

[JohnSully]

Hi @botzill I have this as blocking for v5.3.3 which I'm trying to release ASAP. Should be another few days.

[JohnSully]

Repros with the following python script. Can't get it to repro just playing around on the CLI.

import redis
from random import random

r = redis.Redis(host='localhost', port=6379, db=0)

while True:
    key = "key_" + str(int(random() * 10000))
    subkey = "key_" + str(int(random() * 10000))
    if (random() >= 0.5):
        r.sadd(key, subkey)
        r.execute_command('expiremember ' + key + ' ' + subkey + ' 10')
    else:
        r.delete(key)

[JohnSully]

This is fixed in change b357803

After a subkey expiration is performed but when there are still remaining subkey expires remaining we need to resort the expireset.

The change will be in v5.3.3 scheduled to be released ASAP.

[botzill]

thx @JohnSully. Waiting to test this.