Accelio thread stuck in busy loop

[redlicha]

On multiple nodes (g8-4, RDMA enabled) it was observed that the xio thread went into a busy loop:

• stor-08

(gdb) t 121
[Switching to thread 121 (Thread 0x7f1da37fe700 (LWP 6324))]
#0  0x00007f1ebb083b7e in xio_nexus_release_cb (data=<optimized out>) at ../common/xio_nexus.c:1096
1096	../common/xio_nexus.c: No such file or directory.
(gdb) bt
#0  0x00007f1ebb083b7e in xio_nexus_release_cb (data=<optimized out>) at ../common/xio_nexus.c:1096
#1  0x00007f1ebb053eed in xio_ev_loop_exec_scheduled (loop=loop@entry=0x7f1d9400fa20) at xio/xio_ev_loop.c:368
#2  0x00007f1ebb053f83 in xio_ev_loop_run_helper (loop_hndl=0x7f1d9400fa20, timeout=timeout@entry=-1) at xio/xio_ev_loop.c:412
#3  0x00007f1ebb0542fa in xio_ev_loop_run (loop_hndl=<optimized out>) at xio/xio_ev_loop.c:514
#4  0x00007f1ebb0567b5 in xio_context_run_loop (ctx=0x7f1d940082f0, timeout_ms=timeout_ms@entry=-1) at xio/xio_context.c:504
#5  0x0000000000575db6 in volumedriverfs::NetworkXioServer::run (this=0x297d6f8, promise=...)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/NetworkXioServer.cpp:352
#6  0x00000000005781c8 in volumedriverfs::NetworkXioInterface::run (this=<optimized out>, promise=...)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/NetworkXioInterface.cpp:38
#7  0x000000000052c3d6 in volumedriverfs::FuseInterface::<lambda()>::operator() (__closure=<optimized out>)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/FuseInterface.cpp:357
#8  boost::detail::thread_data<volumedriverfs::FuseInterface::operator()(const boost::filesystem::path&, const std::vector<std::__cxx11::basic_string<char> >&)::<lambda()> >::run(void) (
    this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
#9  0x00007f1eb85a85d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#10 0x00007f1ebae3870a in start_thread (arg=0x7f1da37fe700) at pthread_create.c:333
#11 0x00007f1eb584982d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

• stor-05:

(gdb) t 131
[Switching to thread 131 (Thread 0x7fe15bfff700 (LWP 10981))]
#0  0x00007fe374969e89 in INIT_LIST_HEAD (list=<optimized out>) at ./linux/list.h:59
59	./linux/list.h: No such file or directory.
(gdb) bt
#0  0x00007fe374969e89 in INIT_LIST_HEAD (list=<optimized out>) at ./linux/list.h:59
#1  list_del_init (entry=<optimized out>) at ./linux/list.h:166
#2  xio_ev_loop_remove_event (evt=0x7fda44f4fb38) at xio/xio_ev_loop.c:332
#3  0x00007fe374969ee7 in xio_ev_loop_exec_scheduled (loop=loop@entry=0x7fe11c009810) at xio/xio_ev_loop.c:361
#4  0x00007fe374969f83 in xio_ev_loop_run_helper (loop_hndl=0x7fe11c009810, timeout=timeout@entry=-1) at xio/xio_ev_loop.c:412
#5  0x00007fe37496a2fa in xio_ev_loop_run (loop_hndl=<optimized out>) at xio/xio_ev_loop.c:514
#6  0x00007fe37496c7b5 in xio_context_run_loop (ctx=0x7fe11c0095f0, timeout_ms=timeout_ms@entry=-1) at xio/xio_context.c:504
#7  0x0000000000575db6 in volumedriverfs::NetworkXioServer::run (this=0x2acb568, promise=...)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/NetworkXioServer.cpp:352
#8  0x00000000005781c8 in volumedriverfs::NetworkXioInterface::run (this=<optimized out>, promise=...)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/NetworkXioInterface.cpp:38
#9  0x000000000052c3d6 in volumedriverfs::FuseInterface::<lambda()>::operator() (__closure=<optimized out>)
    at /opt/vmachines/jenkins_work/workspace/volumedriver-no-dedup-alba-message-builder-release-ubuntu-16.04/volumedriver-core/src/filesystem/FuseInterface.cpp:357
#10 boost::detail::thread_data<volumedriverfs::FuseInterface::operator()(const boost::filesystem::path&, const std::vector<std::__cxx11::basic_string<char> >&)::<lambda()> >::run(void) (
    this=<optimized out>) at /usr/include/boost/thread/detail/thread.hpp:116
#11 0x00007fe371ebe5d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#12 0x00007fe37474e6fa in start_thread (arg=0x7fe15bfff700) at pthread_create.c:333
#13 0x00007fe36f15fb5d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109