New invalid argument error in libzmq, mutex.hpp:142

[nightlark]

Issue description

One of the unit test suites we have finishes all of the test cases (which involve starting up and shutting down zmq, probably at least 50 times), but then there is an invalid argument error by zmq after boost reports the number of test cases that have passed. The first time observing this issue was on March 9th, but we cache builds of dependencies. The cached version on our develop branch was last updated February 27. Switching our builds to use the last tagged release (v4.2.3) makes the error go away.

The output for the zmq error: Invalid argument (/home/travis/build/GMLC-TDC/HELICS-src/libzmq/src/mutex.hpp:142)

Environment

Travis CI Linux VMs using gcc-6, gcc-4.9, and clang-3.6

• libzmq version (commit hash if unreleased): One of the commits between February 27 and March 9.
• OS: Linux

Minimal test code / Steps to reproduce the issue

1. Checkout and build GMLC-TDC/HELICS-src develop branch
2. Run key-tests test suite. I haven't narrowed it down to a specific zeromq commit, or line in our test suite. If that changes I'll provide an update here.

What's the actual result? (include assertion message & call stack if applicable)

https://travis-ci.org/GMLC-TDC/HELICS-src/jobs/353188544 - see error at the bottom

What's the expected result?

https://travis-ci.org/GMLC-TDC/HELICS-src/jobs/353393510

[bluca]

It's fixed in the latest master

[bluca]

@nightlark did you have any chance to try the latest master?

[nightlark]

I tried the latest commit that was made to master yesterday, and it still had the issue. I'm rerunning the build with the commit made today.

[bluca]

Today's changes won't affect that. Could you try with this commit? 9bd2d3f

[nightlark]

I'm running a build with that commit now. Pretty much what I found yesterday from the merged PR commits is:

2960 - works
2967 - works
2970 - works
2971 - doesn't work
2973 - doesn't work
2974 - works (merge for commit 9bd2d3f)
2976 - doesn't work (merge for reverting 2974)
2956 - doesn't work

[sigiesec]

@bluca You probably assume that this is related to the mutex in random.cpp, right? Maybe that isn't the case at all. It would be good to have some more information, i.e. the stack traces of all threads when the assertion occurs.

[nightlark]

Commit 9bd2d3f works. I need to figure out how to get a stack trace out of travis.

[sigiesec]

Okay, then it probably is related to the destruction order somehow. I had a look at where context_t is used in your code, but did not find something suspicous after giving it a quick look.
I think it will be quite complex to get a stack trace out of travis.

Maybe you can reproduce it locally within gdb?

[nightlark]

I'll see if I can update my Linux vm and get a stack trace later today or tomorrow.

[nightlark]

I was able to reproduce it locally within gdb.

Program received signal SIGABRT, Aborted.
0x00002aaaacb9e1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

Thread 2 (Thread 0x2aaaad12c700 (LWP 158139)):

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00002aaaac1dc15c in __gthread_cond_wait (__mutex=<optimized out>, __cond=<optimized out>)
    at /tmp/mockbuild/spack-stage/spack-stage-P3jF0z/spack-build/x86_64-unknown-linux-gnu/libstdc++-v3/include/x86_64-unknown-linux-gnu/bits/gthr-default.h:864
#2  std::condition_variable::wait (this=<optimized out>, __lock=...)
    at /builddir/build/BUILD/gccspack/spack/var/spack/stage/gcc-4.9.3-34vmmvov5ct2riknhkfcmxieffb5pyex/gcc-4.9.3/libstdc++-v3/src/c++11/condition_variable.cc:52
#3  0x000000000062a1f4 in BlockingQueue<std::pair<int, std::string> >::pop() ()
#4  0x00000000006280e4 in helics::LoggingCore::processingLoop() ()
#5  0x00000000006309bb in void std::_Mem_fn<void (helics::LoggingCore::*)()>::operator()<, void>(helics::LoggingCore*) const ()
#6  0x00000000006308cf in void std::_Bind_simple<std::_Mem_fn<void (helics::LoggingCore::*)()> (helics::LoggingCore*)>::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#7  0x000000000063076d in std::_Bind_simple<std::_Mem_fn<void (helics::LoggingCore::*)()> (helics::LoggingCore*)>::operator()()
    ()
#8  0x0000000000630686 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (helics::LoggingCore::*)()> (helics::LoggingCore*)> >::_M_run() ()
#9  0x00002aaaac1dfdc0 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>)
    at /builddir/build/BUILD/gccspack/spack/var/spack/stage/gcc-4.9.3-34vmmvov5ct2riknhkfcmxieffb5pyex/gcc-4.9.3/libstdc++-v3/src/c++11/thread.cc:84
#10 0x00002aaaac954e25 in start_thread (arg=0x2aaaad12c700) at pthread_create.c:308
#11 0x00002aaaacc6134d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Thread 1 (Thread 0x2aaaaab19940 (LWP 158132)):

#0  0x00002aaaacb9e1f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00002aaaacb9f8e8 in __GI_abort () at abort.c:90
#2  0x00002aaaab2082be in zmq::zmq_abort(char const*) () from /g/g19/usrname/helics/zmq-dep/lib64/libzmq.so.5
#3  0x00002aaaab21f3cc in zmq::random_close() () from /g/g19/usrname/helics/zmq-dep/lib64/libzmq.so.5
#4  0x00002aaaab1fae52 in zmq::ctx_t::~ctx_t() () from /g/g19/usrname/helics/zmq-dep/lib64/libzmq.so.5
#5  0x00002aaaab1fb4e9 in zmq::ctx_t::terminate() () from /g/g19/usrname/helics/zmq-dep/lib64/libzmq.so.5
#6  0x00002aaaab2481ad in zmq_ctx_term () from /g/g19/usrname/helics/zmq-dep/lib64/libzmq.so.5
#7  0x0000000000646027 in zmq::context_t::close() ()
#8  0x0000000000645ffc in zmq::context_t::~context_t() ()
#9  0x0000000000646940 in std::default_delete<zmq::context_t>::operator()(zmq::context_t*) const ()
#10 0x00000000006462f9 in std::unique_ptr<zmq::context_t, std::default_delete<zmq::context_t> >::~unique_ptr() ()
#11 0x0000000000645dfe in zmqContextManager::~zmqContextManager() ()
#12 0x0000000000645e3e in zmqContextManager::~zmqContextManager() ()
#13 0x0000000000647716 in std::_Sp_counted_ptr<zmqContextManager*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
#14 0x00000000004197d0 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#15 0x00000000004181cf in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ()
#16 0x00000000006121b0 in std::__shared_ptr<zmqContextManager, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() ()
#17 0x00000000006121ca in std::shared_ptr<zmqContextManager>::~shared_ptr() ()
#18 0x0000000000647612 in std::pair<std::string const, std::shared_ptr<zmqContextManager> >::~pair() ()
#19 0x000000000064763c in void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >::destroy<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >(std::pair<std::string const, std::shared_ptr<zmqContextManager> >*) ()
#20 0x000000000064748d in std::enable_if<std::__and_<std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > > >::__destroy_helper<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >::type>::value, void>::type std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > > >::_S_destroy<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >(std::allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >&, std::pair<std::string const, std::shared_ptr<zmqContextManager> >*) ()
#21 0x0000000000647398 in void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > > >::destroy<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >(std::allocator<std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >&, std::pair<std::string const, std::shared_ptr<zmqContextManager> >*) ()
#22 0x0000000000647095 in std::_Rb_tree<std::string, std::pair<std::string const, std::shared_ptr<zmqContextManager> >, std::_Select1st<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >::_M_destroy_node(std::_Rb_tree_node<std::pair<std::string const, std::sha---Type <return> to continue, or q <return> to quit---
red_ptr<zmqContextManager> > >*) ()
#23 0x0000000000646b1d in std::_Rb_tree<std::string, std::pair<std::string const, std::shared_ptr<zmqContextManager> >, std::_Select1st<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >*) ()
#24 0x00000000006464da in std::_Rb_tree<std::string, std::pair<std::string const, std::shared_ptr<zmqContextManager> >, std::_Select1st<std::pair<std::string const, std::shared_ptr<zmqContextManager> > >, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >::~_Rb_tree() ()
#25 0x00000000006476dc in std::map<std::string, std::shared_ptr<zmqContextManager>, std::less<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<zmqContextManager> > > >::~map() ()
#26 0x00002aaaacba1a69 in __run_exit_handlers (status=0, listp=0x2aaaacf256c8 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true) at exit.c:77
#27 0x00002aaaacba1ab5 in __GI_exit (status=<optimized out>) at exit.c:99
#28 0x00002aaaacb8ac0c in __libc_start_main (main=0x4144b5 <main>, argc=1, ubp_av=0x7fffffffbda8, init=<optimized out>,
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffbd98) at ../csu/libc-start.c:308
#29 0x0000000000414369 in _start ()

[bluca]

What's zmqContextManager ?

[nightlark]

It's a singleton wrapper around a pointer to the zmq context. Maybe there's some other things that should be getting done before the zmq context is released?

https://github.com/GMLC-TDC/HELICS-src/blob/master/src/helics/common/zmqContextManager.cpp
https://github.com/GMLC-TDC/HELICS-src/blob/master/src/helics/common/zmqContextManager.h