keydb-server v5.2.0 very slow

[rnz]

KeyDB v5.1.1 (with UseASM=0) to day is crash after 4 days and after 2h (UTC):

[Tue Dec 31 13:19:00 2019] keydb-server[408043]: segfault at 0 ip 000055b1aa505258 sp 00007f8e6edf92a0 error 4 in keydb-server[55b1aa468000+1da000]
[Tue Dec 31 13:46:36 2019] keydb-server[1509667]: segfault at 7f51eb713c17 ip 00007f516c1f8a18 sp 00007f516c1f8a20 error 6
[Tue Dec 31 16:18:17 2019] keydb-server[1041063]: segfault at 0 ip 000055a914203258 sp 00007f77543f92a0 error 4 in keydb-server[55a914166000+1da000]

After last crush I was upgrade keydb to v5.2.0 and saw very slow performance (UTC+3):

Happy New Year! ;)

[JohnSully]

Thanks for the feedback. There was a tweak to the spinlock changing the maximum spin period prior to sleeping.

I will get you a build with the old behavior to try.

[JohnSully]

Offending change: 8a9045c#diff-7370d7f639c5200989abebe77242bf30

Note that the addition of mfence in that change is required so the fix is not a simple revert.

[JohnSully]

Change dbef17c has been committed. Please let me know if this resolves your issues. If you use docker the unstable image rebuilds nightly so should be available for you to try tomorrow.

[rnz]

additional info about crash of v5.1.1 (UseASM=false):

1629985:C 01 Jan 2020 15:25:27.618 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1629985:C 01 Jan 2020 15:25:27.618 # KeyDB version=5.1.1, bits=64, commit=a91f58b8, modified=1, pid=1629985, just started
1629985:C 01 Jan 2020 15:25:27.618 # Configuration loaded
1629986:M 01 Jan 2020 15:25:27.620 * Increased maximum number of open files to 65032 (it was originally set to 1024).
                                                                     
                                                                     
                                        KeyDB 5.1.1 (a91f58b8/1) 64 bit
                                                                     
                                        Running in standalone mode
                                        Port: 6379
                                        PID: 1629986
                                                                     
                     Like KeyDB? Star us on GitHub!                  
                                                                     
                     https://github.com/JohnSully/KeyDB              
                                                                     

1629986:M 01 Jan 2020 15:25:27.622 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1629986:M 01 Jan 2020 15:25:27.622 # Server initialized
1629986:M 01 Jan 2020 15:25:27.622 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with KeyDB. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. KeyDB must be restarted after THP is disabled.
1629986:M 01 Jan 2020 15:25:27.622 * Loading RDB produced by version 5.1.1
1629986:M 01 Jan 2020 15:25:27.622 * RDB age 2006 seconds
1629986:M 01 Jan 2020 15:25:27.623 * RDB memory usage when created 4475.99 Mb
1629986:M 01 Jan 2020 15:25:42.857 * DB loaded from disk: 15.236 seconds
1629986:M 01 Jan 2020 15:25:42.857 * Ready to accept connections
1629986:M 01 Jan 2020 15:25:42.858   Thread 0 alive.
1629986:M 01 Jan 2020 15:25:42.858   Thread 1 alive.
1629986:M 01 Jan 2020 15:25:42.858   Thread 2 alive.
1629986:M 01 Jan 2020 15:25:42.858   Thread 3 alive.
1629986:M 01 Jan 2020 16:37:18.902 * 5000000 changes in 900 seconds. Saving...
1629986:M 01 Jan 2020 16:37:19.045 * Background saving started by pid 1643999
1643999:C 01 Jan 2020 16:37:47.484 * DB saved on disk
1643999:C 01 Jan 2020 16:37:47.777 * RDB: 116 MB of memory used by copy-on-write
1629986:M 01 Jan 2020 16:37:47.917 * Background saving terminated with success
1629986:M 01 Jan 2020 17:52:42.288 * 5000000 changes in 900 seconds. Saving...
1629986:M 01 Jan 2020 17:52:42.467 * Background saving started by pid 1658220
1658220:C 01 Jan 2020 17:53:10.946 * DB saved on disk
1658220:C 01 Jan 2020 17:53:11.401 * RDB: 115 MB of memory used by copy-on-write
1629986:M 01 Jan 2020 17:53:11.567 * Background saving terminated with success
1629986:M 01 Jan 2020 18:01:33.101 # Out Of Memory allocating 155564 bytes!


=== KEYDB BUG REPORT START: Cut & paste starting from here ===
1629986:M 01 Jan 2020 18:01:33.102 # ------------------------------------------------
1629986:M 01 Jan 2020 18:01:33.102 # !!! Software Failure. Press left mouse button to continue
1629986:M 01 Jan 2020 18:01:33.102 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.cpp:4857
1629986:M 01 Jan 2020 18:01:33.102 # (forcing SIGSEGV in order to print the stack trace)
1629986:M 01 Jan 2020 18:01:33.102 # ------------------------------------------------
1629986:M 01 Jan 2020 18:01:33.102 # KeyDB 5.1.1 crashed by signal: 11
1629986:M 01 Jan 2020 18:01:33.102 # Crashed running the instruction at: 0x55d53415b40c
1629986:M 01 Jan 2020 18:01:33.102 # Accessing address: 0xffffffffffffffff
1629986:M 01 Jan 2020 18:01:33.102 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/local/bin/keydb-server 0.0.0.0:6379(_serverPanic+0x12c) [0x55d53415b40c]

Backtrace:
/usr/local/bin/keydb-server 0.0.0.0:6379(logStackTrace(ucontext*)+0x32) [0x55d53415c472]
/usr/local/bin/keydb-server 0.0.0.0:6379(sigsegvHandler(int, siginfo_t*, void*)+0xc8) [0x55d53415cb98]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110e0) [0x7f13923030e0]
/usr/local/bin/keydb-server 0.0.0.0:6379(_serverPanic+0x12c) [0x55d53415b40c]
/usr/local/bin/keydb-server 0.0.0.0:6379(redisOutOfMemoryHandler(unsigned long)+0x34) [0x55d534104044]
/usr/local/bin/keydb-server 0.0.0.0:6379(zmalloc+0x49) [0x55d53410f0c9]
/usr/local/bin/keydb-server 0.0.0.0:6379(sdsnewlen+0x61) [0x55d53410c521]
/usr/local/bin/keydb-server 0.0.0.0:6379(processMultibulkBuffer(client*)+0x344) [0x55d534118d94]
/usr/local/bin/keydb-server 0.0.0.0:6379(processInputBuffer(client*, int)+0x146) [0x55d5341192c6]
/usr/local/bin/keydb-server 0.0.0.0:6379(readQueryFromClient(aeEventLoop*, int, void*, int)+0x330) [0x55d53411bc00]
/usr/local/bin/keydb-server 0.0.0.0:6379(ProcessEventCore+0x176) [0x55d5340ff826]
/usr/local/bin/keydb-server 0.0.0.0:6379(aeProcessEvents+0x145) [0x55d5340ffa05]
/usr/local/bin/keydb-server 0.0.0.0:6379(aeMain+0x35) [0x55d5340ffd95]
/usr/local/bin/keydb-server 0.0.0.0:6379(workerThreadMain(void*)+0x97) [0x55d534104117]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4) [0x7f13922f94a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f139203bd0f]

------ INFO OUTPUT ------
# Server
redis_version:5.1.1
redis_git_sha1:a91f58b8
redis_git_dirty:1
redis_build_id:f8c7776a6b2fa5be
redis_mode:standalone
os:Linux 4.15.18-9-pve x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1629986
run_id:c802b3c8781002f6fcea2ba712de5435e5d1cae7
tcp_port:6379
uptime_in_seconds:9366
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:843389
executable:/usr/local/bin/keydb-server
config_file:/etc/keydb/redis.conf

# Clients
connected_clients:1244
client_recent_max_input_buffer:49408
client_recent_max_output_buffer:0
blocked_clients:0
current_client_thread:3
thread_0_clients:288
thread_1_clients:294
thread_2_clients:318
thread_3_clients:344

# Memory
used_memory:5009216000
used_memory_human:4.67G
used_memory_rss:5375877120
used_memory_rss_human:5.01G
used_memory_peak:5049662376
used_memory_peak_human:4.70G
used_memory_peak_perc:99.20%
used_memory_overhead:129406192
used_memory_startup:9125824
used_memory_dataset:4879809808
used_memory_dataset_perc:97.59%
allocator_allocated:5010000360
allocator_active:5593829376
allocator_resident:5661523968
total_system_memory:2151759650816
total_system_memory_human:1.96T
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.12
allocator_frag_bytes:583829016
allocator_rss_ratio:1.01
allocator_rss_bytes:67694592
rss_overhead_ratio:0.95
rss_overhead_bytes:-285646848
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:366596976
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:36417384
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:585562
rdb_bgsave_in_progress:0
rdb_last_save_time:1577901191
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:29
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:120922112
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:7021
total_commands_processed:56592248
instantaneous_ops_per_sec:5629
total_net_input_bytes:31905197748
total_net_output_bytes:4183993951
instantaneous_input_kbps:2539.09
instantaneous_output_kbps:341.20
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:19879
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:40741014
keyspace_misses:5258770
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:178436
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:d909b2908fbb37a54d4d1f2416d15028eaf0b074
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:2064.866506
used_cpu_user:947.512707
used_cpu_sys_children:10.660739
used_cpu_user_children:43.536373
server_threads:4
long_lock_waits:14860735

# Commandstats
cmdstat_select:calls=1211,usec=1268,usec_per_call=1.05
cmdstat_set:calls=112683,usec=1012754,usec_per_call=8.99
cmdstat_smembers:calls=14902988,usec=100310665,usec_per_call=6.73
cmdstat_hmset:calls=10360323,usec=125539113,usec_per_call=12.12
cmdstat_expire:calls=112683,usec=400084,usec_per_call=3.55
cmdstat_get:calls=82453,usec=836185,usec_per_call=10.14
cmdstat_hget:calls=24882566,usec=91189859,usec_per_call=3.66
cmdstat_info:calls=5564,usec=2820273,usec_per_call=506.88
cmdstat_hgetall:calls=6131778,usec=63030667,usec_per_call=10.28

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1264330,expires=355758,avg_ttl=50247661

------ CLIENT LIST OUTPUT ------
1629986:M 01 Jan 2020 18:01:33.106 # Out Of Memory allocating 248810 bytes!
1629986:M 01 Jan 2020 18:01:33.106 # ------------------------------------------------
1629986:M 01 Jan 2020 18:01:33.106 # !!! Software Failure. Press left mouse button to continue
1629986:M 01 Jan 2020 18:01:33.106 # Guru Meditation: Redis aborting for OUT OF MEMORY #server.cpp:4857
1629986:M 01 Jan 2020 18:01:33.106 # (forcing SIGSEGV in order to print the stack trace)
1629986:M 01 Jan 2020 18:01:33.106 # ------------------------------------------------

"Out of memory" is wrong - container LXC have:

# free -m
              total        used        free      shared  buff/cache   available
Mem:         102400        5184       97145          17          69       97215
Swap:             0           0           0

# df -h
Filesystem               Size  Used Avail Use% Mounted on
pool1/subvol-311-disk-0  150G  3.0G  148G   2% /
none                     492K     0  492K   0% /dev
tmpfs                   1002G     0 1002G   0% /dev/shm
tmpfs                   1002G   17M 1002G   1% /run
tmpfs                    5.0M     0  5.0M   0% /run/lock
tmpfs                   1002G     0 1002G   0% /sys/fs/cgroup

# prlimit -p $(pgrep keydb-server)
RESOURCE   DESCRIPTION                             SOFT      HARD UNITS
AS         address space limit                unlimited unlimited bytes
CORE       max core file size                         0 unlimited bytes
CPU        CPU time                           unlimited unlimited seconds
DATA       max data size                      unlimited unlimited bytes
FSIZE      max file size                      unlimited unlimited bytes
LOCKS      max number of file locks held      unlimited unlimited locks
MEMLOCK    max locked-in-memory address space     65536     65536 bytes
MSGQUEUE   max bytes in POSIX mqueues            819200    819200 bytes
NICE       max nice prio allowed to raise             0         0 
NOFILE     max number of open files               65032     65032 files
NPROC      max number of processes              8207883   8207883 processes
RSS        max resident set size              unlimited unlimited bytes
RTPRIO     max real-time priority                     0         0 
RTTIME     timeout for real-time tasks        unlimited unlimited microsecs
SIGPENDING max number of pending signals        8207883   8207883 signals
STACK      max stack size                       8388608 unlimited bytes

And config:

# egrep -v '^#|^$' /etc/keydb/redis.conf 
bind 0.0.0.0
protected-mode no
port 6379
tcp-backlog 511
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /var/run/redis_6379.pid
loglevel notice
logfile "/var/log/keydb-server.log"
databases 16
always-show-logo yes
save 900 5000000
save 300 100000000
save 60 1000000000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /root/
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
replica-priority 100
maxclients 65000
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
replica-lazy-flush no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
server-threads 4

[rnz]

Change dbef17c has been committed. Please let me know if this resolves your issues. If you use docker the unstable image rebuilds nightly so should be available for you to try tomorrow.

Thank you! I will try this version after holidays.

[JohnSully]

I'm going to close this as the fix shipped in 5.3.0, let me know if you are still seeing issues and I will reopen.

[rnz]

I was test 5.3.0 it is slow as 5.2.0:

5.1.1 (UseASM=false) before 13:54
5.3.0 - 13:58 - 13:59
5.1.1 (UseASM=false) after 14:08

1114017:C 15 Jan 2020 10:55:01.422 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1114017:C 15 Jan 2020 10:55:01.422 # KeyDB version=5.3.0, bits=64, commit=39acb312, modified=0, pid=1114017, just started
1114017:C 15 Jan 2020 10:55:01.422 # Configuration loaded
1114018:M 15 Jan 2020 10:55:01.425 * Increased maximum number of open files to 65032 (it was originally set to 1024).
                                                                     
                                                                     
                                        KeyDB 5.3.0 (39acb312/0) 64 bit
                                                                     
                                        Running in standalone mode
                                        Port: 6379
                                        PID: 1114018
                                                                     
                     Like KeyDB? Star us on GitHub!                  
                                                                     
                     https://github.com/JohnSully/KeyDB              
                                                                     

1114018:M 15 Jan 2020 10:55:01.427 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1114018:M 15 Jan 2020 10:55:01.427 # Server initialized
1114018:M 15 Jan 2020 10:55:01.427 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with KeyDB. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. KeyDB must be restarted after THP is disabled.
1114018:M 15 Jan 2020 10:55:01.428 * Loading RDB produced by version 5.1.1
1114018:M 15 Jan 2020 10:55:01.428 * RDB age 73 seconds
1114018:M 15 Jan 2020 10:55:01.428 * RDB memory usage when created 5822.16 Mb
1114018:M 15 Jan 2020 10:55:21.153 * DB loaded from disk: 19.726 seconds
1114018:M 15 Jan 2020 10:55:21.153 * Ready to accept connections
1114018:M 15 Jan 2020 10:55:21.154   Thread 0 alive.
1114018:M 15 Jan 2020 10:55:21.154   Thread 1 alive.
1114018:M 15 Jan 2020 10:55:21.154   Thread 2 alive.
1114018:M 15 Jan 2020 10:55:21.160   Thread 3 alive.
1114018:signal-handler (1579085968) Received SIGTERM scheduling shutdown...
1114018:M 15 Jan 2020 10:59:28.250 # User requested shutdown...
1114018:M 15 Jan 2020 10:59:28.251 * Saving the final RDB snapshot before exiting.
1114018:M 15 Jan 2020 11:00:02.867 * DB saved on disk
1114018:M 15 Jan 2020 11:00:02.868 * Removing the pid file.
1114018:M 15 Jan 2020 11:00:02.868 # KeyDB is now ready to exit, bye bye...
1114841:C 15 Jan 2020 11:01:01.514 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1114841:C 15 Jan 2020 11:01:01.514 # KeyDB version=5.1.1, bits=64, commit=a91f58b8, modified=1, pid=1114841, just started
1114841:C 15 Jan 2020 11:01:01.514 # Configuration loaded
1114842:M 15 Jan 2020 11:01:01.517 * Increased maximum number of open files to 65032 (it was originally set to 1024).

# keydb-cli info
# Server
redis_version:5.3.0
redis_git_sha1:39acb312
redis_git_dirty:0
redis_build_id:3dd01d75dd627cb1
redis_mode:standalone
os:Linux 4.15.18-9-pve x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1114018
run_id:d6dee688b81767a4a87f5a67f36e8c9528ed9fea
tcp_port:6379
uptime_in_seconds:58
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2027455
executable:/usr/local/bin/keydb-server
config_file:/etc/keydb/redis.conf

# Clients
connected_clients:767
client_recent_max_input_buffer:359372
client_recent_max_output_buffer:20504
blocked_clients:0
current_client_thread:2
thread_0_clients:159
thread_1_clients:218
thread_2_clients:199
thread_3_clients:191

# Memory
used_memory:6092647152
used_memory_human:5.67G
used_memory_rss:6387240960
used_memory_rss_human:5.95G
used_memory_peak:6098278392
used_memory_peak_human:5.68G
used_memory_peak_perc:99.91%
used_memory_overhead:136670308
used_memory_startup:9127360
used_memory_dataset:5955976844
used_memory_dataset_perc:97.90%
allocator_allocated:6094874088
allocator_active:6723264512
allocator_resident:6791831552
total_system_memory:2151759650816
total_system_memory_human:1.96T
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.10
allocator_frag_bytes:628390424
allocator_rss_ratio:1.01
allocator_rss_bytes:68567040
rss_overhead_ratio:0.94
rss_overhead_bytes:-404590592
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:294021928
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:34907340
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:9486
rdb_bgsave_in_progress:0
rdb_last_save_time:1579085701
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:812
total_commands_processed:58222
instantaneous_ops_per_sec:1132
total_net_input_bytes:134784701
total_net_output_bytes:5590167
instantaneous_input_kbps:2017.38
instantaneous_output_kbps:110.43
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:298
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:42651
keyspace_misses:5509
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:89543cffce27dfd04f0269f9612ebda7d265221c
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:5.493320
used_cpu_user:79.956979
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
server_threads:4
long_lock_waits:41190

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1403673,expires=483802,avg_ttl=50766207