PUB crash when SUB exceeded SNDHWM

[sublee]

Please use this template for reporting suspected bugs or requests for help.

Issue description

When all of these conditions are satisfied, the assertion failure from mtrie.cpp occurs:

• A connection between a PUB socket and many SUB sockets.
• A SUB socket subscribe/unsubscribe many prefixes.
• Call zmq_getsockopt() with ZMQ_EVENTS for SUB sockets.

Assertion failed: erased == 1 (src/mtrie.cpp:297)
[1]    30266 abort (core dumped)  ./a.out

Environment

• libzmq version (commit hash if unreleased): 4.2.0 and 4.2.3
• OS: Ubuntu 16.04 LTS

Minimal test code / Steps to reproduce the issue

To reproduce this crash, we should prepare a PUB socket and many SUB sockets.

We will call this sequence (pseudo-code): pub.connect(sub) or sub.connect(pub); pub.getsockopt(ZMQ_EVENTS); sub.subscribe(prefix); sub.getsockopt(ZMQ_EVENTS); sub.unsubscribe(prefix); sub.getsockopt(ZMQ_EVENTS). There will be many prefixes to subscribe/unsubscribe.

Calling getsockopt(ZMQ_EVENTS) after SUB's SUBSCRIBE/UNSUBSCRIBE, or PUB's zmq_connect() will produce a crash due to the assertion failure in mtrie_t::rm_helper.

You can switch PUB<->SUB connection topology by the pub_to_sub variable.

#include "zmq.h"
#include <stdio.h>

// Set 1 or 0 to switch the PUB<->SUB connection topology.
static int pub_to_sub = 1;

void gen_topic(int n, char* topic)
{
    // Simple hash function to generate a subscription prefix from a number.
    n = (n * 2654435761);
    sprintf(topic, "%08x", n);
}

void getsockopt_events_within_many_subscriptions(void* sub)
{
    char topic[8];
    char opt[256];
    size_t opt_len = 256;

    for (int j = 0; j < 10000; ++j)
    {
        gen_topic(j, topic);
        zmq_setsockopt(sub, ZMQ_SUBSCRIBE, &topic, 8);
        // CRASH: Get ZMQ_EVENTS from a SUB socket.
        zmq_getsockopt(sub, ZMQ_EVENTS, opt, &opt_len);
    }
    for (int j = 0; j < 10000; ++j)
    {
        gen_topic(j, topic);
        zmq_setsockopt(sub, ZMQ_UNSUBSCRIBE, &topic, 8);
        // CRASH: Get ZMQ_EVENTS from a SUB socket.
        zmq_getsockopt(sub, ZMQ_EVENTS, opt, &opt_len);
    }
}

int main()
{
    printf("%d.%d.%d\n", ZMQ_VERSION_MAJOR, ZMQ_VERSION_MINOR, ZMQ_VERSION_PATCH);

    void *context = zmq_ctx_new();
    void *pub = zmq_socket(context, ZMQ_PUB);
    void *sub;

    char addr[256]; size_t addr_len = 256;
    char opt[256];  size_t opt_len  = 256;

    if (pub_to_sub)
    {
        // PUB->SUB
        for (int i = 0; i < 100; ++i)
        {
            sub = zmq_socket(context, ZMQ_SUB);

            zmq_bind(sub, "tcp://127.0.0.1:*");
            zmq_getsockopt(sub, ZMQ_LAST_ENDPOINT, addr, &addr_len);
            zmq_connect(pub, addr);

            getsockopt_events_within_many_subscriptions(sub);
        }
    }
    else
    {
        // SUB->PUB
        zmq_bind(pub, "tcp://127.0.0.1:*");
        zmq_getsockopt(pub, ZMQ_LAST_ENDPOINT, addr, &addr_len);
        for (int i = 0; i < 100; ++i)
        {
            sub = zmq_socket(context, ZMQ_SUB);

            zmq_connect(sub, addr);

            getsockopt_events_within_many_subscriptions(sub);

            // CRASH: Get ZMQ_EVENTS from the PUB socket.
            zmq_getsockopt(pub, ZMQ_EVENTS, opt, &opt_len);
        }
    }
}

What's the actual result? (include assertion message & call stack if applicable)

$ gcc zmq_events_crash.c -L ~/usr/local/lib -lzmq && ./a.out
4.2.3
Assertion failed: erased == 1 (src/mtrie.cpp:297)
[1]    30266 abort (core dumped)  ./a.out

What's the expected result?

$ gcc zmq_events_crash.c -L ~/usr/local/lib -lzmq && ./a.out
4.2.3
$ echo $?
0

When SUB sockets connect to the PUB socket, this crash doesn't happen.

[bluca]

I can't seem to reproduce the crash. Did you build the library with or without draft apis?

[sublee]

@bluca Oh... I built libzmq-4.2.3 without any configuration arguments.

I guess low SNDHWM on SUB sockets reproduce the crash easier. Would you please try it?

zmq_setsockopt(sub, ZMQ_SNDHWM, (void *)10, 0);

[sublee]

Here's my gdb backtrace:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
4.2.3
[New Thread 0x7ffff6abc700 (LWP 23970)]
[New Thread 0x7ffff62bb700 (LWP 23972)]
Assertion failed: erased == 1 (src/mtrie.cpp:297)

Thread 1 "a.out" received signal SIGABRT, Aborted.
0x00007ffff77b8428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff77b8428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007ffff77ba02a in __GI_abort () at abort.c:89
#2  0x00007ffff7b73539 in zmq::zmq_abort (errmsg_=errmsg_@entry=0x7ffff7bb75c6 "erased == 1") at src/err.cpp:87
#3  0x00007ffff7b7de6f in zmq::mtrie_t::rm_helper (this=this@entry=0xa34380, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:297
#4  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0xa34360, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#5  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0xa34340, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#6  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0xa34320, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#7  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0xa34300, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#8  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0x61c7c0, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#9  0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0x960c50, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#10 0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0x61f020, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#11 0x00007ffff7b7dc7b in zmq::mtrie_t::rm_helper (this=this@entry=0x617ce8, prefix_=<optimized out>, size_=<optimized out>, pipe_=0xade9b0) at src/mtrie.cpp:315
#12 0x00007ffff7b7e235 in zmq::mtrie_t::rm (this=this@entry=0x617ce8, prefix_=<optimized out>, size_=<optimized out>, pipe_=<optimized out>) at src/mtrie.cpp:288
#13 0x00007ffff7baddc9 in zmq::xpub_t::xread_activated (this=0x617740, pipe_=0xade9b0) at src/xpub.cpp:115
#14 0x00007ffff7b92f0f in zmq::socket_base_t::process_commands (this=this@entry=0x617740, timeout_=timeout_@entry=0, throttle_=throttle_@entry=false) at src/socket_base.cpp:1378
#15 0x00007ffff7b95829 in zmq::socket_base_t::connect (this=0x617740, addr_=0x7fffffffe0a0 "tcp://127.0.0.1:40429") at src/socket_base.cpp:709
#16 0x0000000000400b53 in main ()

[bluca]

No change. The program just spins forever in one of the getsockopt_events_within_many_subscriptions loops

[bluca]

I am running with libsodium:

$ ldd src/.libs/libzmq.so.5.1.4 | grep sodium
	libsodium.so.18 => /usr/lib/x86_64-linux-gnu/libsodium.so.18 (0x00007f786586e000)

It doesn't make much sense - unless you are using CURVE, and it that example it's not, sodium or tweetnacl will make no difference at all.

Are you sure you didn't have multiple versions of the libraries laying around? And maybe building with one, but running with an old one or viceversa?

[sublee]

I'm sorry my comment about libsodium was hasty. I removed it quickly but you already read.

Here's the comment:

I've checked my own environment again. There was libsodium in my system. When I uninstalled it, the code didn't crash! Can you run the reproduction script after installing libsodium?

[sublee]

Are you sure you didn't have multiple versions of the libraries laying around? And maybe building with one, but running with an old one or viceversa?

Yes, there's only 1 libzmq on my system.

I want to explain the environment difference between us. I downloaded zeromq-4.2.3 from zeromq-4.2.3.tar.gz. I built it with ./configure && make && sudo make install. I ran the script with gcc zmq_events_crash.c -L ~/usr/local/lib -lzmq && ./a.out.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04 LTS
Release:        16.04
Codename:       xenial
$ uname -a
Linux fantine 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[bluca]

You are not running ldconfig - are you sure there are no other libs? What does ldd a.out say?

[sublee]

Oh, I ran ldconfig also. Sorry for omitting it.

The ldd a.out result is:

linux-vdso.so.1 =>  (0x00007ffca17cf000)
libzmq.so.5 => /usr/local/lib/libzmq.so.5 (0x00007f67cdf02000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f67cdb38000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f67cd930000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f67cd713000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f67cd391000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f67cd088000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f67cce72000)
/lib64/ld-linux-x86-64.so.2 (0x00007f67ce18d000)

[sublee]

Anyways, I'm still researching my crash. I found a symptom that PUB socket doesn't receive all of SUBSCRIBE messages by custom logs on zmq::xpub_t::xread_activated(). So some UNSUBSCRIBE message can be processed without it's corresponding SUBSCRIBE. I'll check whether SUB drops outgoing messages or PUB drops incoming messages.

[sublee]

I found the condition. A SUB socket can drop a SUBSCRIBE message due to the not enough SNDHWM. But it can send an UNSUBSCRIBE message for the dropped subscription. Then it kills the connected PUB socket.

If the dropping is a normal behavior of SUB sockets, PUB socket should prevent the crash itself. If it just crashes, we cannot keep an application safely.

[sublee]

#2252 what I reported 14 months ago came to my mind. At that time, you pointed dropped subscriptions due to the HWM. But I couldn't reproduce it without PyZMQ so I tried and gave up to fix it in PyZMQ.

So essentially this issue is same as that issue. But this time, I can reproduce it without PyZMQ. @bluca I want to make it work in your environment also.

I'll replace the subject with "PUB crash when SUB exceeded SNDHWM".

[sublee]

How about this code?

Show code

#include "zmq.h"
#include 
void gen_topic(int n, char* topic)

{

// Simple hash function to generate a subscription prefix from a number.

n = (n * 2654435761);

sprintf(topic, "%08x", n);

}
void getsockopt_events_within_many_subscriptions(void* sub, int i)

{

char topic[8];

char opt[256];

size_t opt_len = 256;
int n = 10000;

for (int j = 0; j < n; ++j)
{
    gen_topic(j, topic);
    zmq_setsockopt(sub, ZMQ_SUBSCRIBE, &topic, 8);
    // CRASH: Get ZMQ_EVENTS from a SUB socket.
    zmq_getsockopt(sub, ZMQ_EVENTS, opt, &opt_len);
}
for (int j = 0; j < n; ++j)
{
    gen_topic(j, topic);
    zmq_setsockopt(sub, ZMQ_UNSUBSCRIBE, &topic, 8);
    // CRASH: Get ZMQ_EVENTS from a SUB socket.
    zmq_getsockopt(sub, ZMQ_EVENTS, opt, &opt_len);
}

}
int main()

{

printf("%d.%d.%d\n", ZMQ_VERSION_MAJOR, ZMQ_VERSION_MINOR, ZMQ_VERSION_PATCH);
void *context = zmq_ctx_new();
void *pub = zmq_socket(context, ZMQ_PUB);
void *sub;

char addr[256]; size_t addr_len = 256;
char opt[256];  size_t opt_len  = 256;
int sub_hwm = 1;

zmq_bind(pub, "tcp://127.0.0.1:*");
zmq_getsockopt(pub, ZMQ_LAST_ENDPOINT, addr, &addr_len);

for (int i = 0; i < 100; ++i)
{
    sub = zmq_socket(context, ZMQ_SUB);
    zmq_setsockopt(sub, ZMQ_SNDHWM, &sub_hwm, sizeof(sub_hwm));

    zmq_connect(sub, addr);
    zmq_getsockopt(pub, ZMQ_EVENTS, opt, &opt_len);

    getsockopt_events_within_many_subscriptions(sub, i);
}

}

Use #2942 (comment) instead.