core: `msg_receive()` on native sometimes returns without `msg` being re-set

[miri64]

Description

Coming from #6123 I was able to track down the issue to the fact that msg_receive() in some corner-cases seems to return without the out parameter being rewritten. I wasn't able to pin-point the exact issue yet and I'm unsure if it is only an issue on native.

Steps to reproduce the issue

I applied the following patch to current master (bdd2d52, might not apply to later versions, please then try to change manually):

diff --git a/sys/net/gnrc/network_layer/ipv6/gnrc_ipv6.c b/sys/net/gnrc/network_layer/ipv6/gnrc_ipv6.c
index feb2e8f10..0c3fa97c4 100644
--- a/sys/net/gnrc/network_layer/ipv6/gnrc_ipv6.c
+++ b/sys/net/gnrc/network_layer/ipv6/gnrc_ipv6.c
@@ -180,6 +180,7 @@ static void *_event_loop(void *args)
     /* start event loop */
     while (1) {
         DEBUG("ipv6: waiting for incoming message.\n");
+        memset(&msg, 0, sizeof(msg));
         msg_receive(&msg);
 
         switch (msg.type) {
@@ -220,6 +221,7 @@ static void *_event_loop(void *args)
                 gnrc_ipv6_nib_handle_timer_event(msg.content.ptr, msg.type);
                 break;
             default:
+                printf("ipv6: unknown message type 0x%04x\n", msg.type);
                 break;
         }
     }

and ran gnrc_networking. I then tried to ping the node as described in #10875:

sudo true; for _ in $(seq 10); do sudo ping -s0 -f "fe80::...%tapbr0" & done

Edit: Alternatively just run the tests in #10908.

Expected results

ipv6: unknown message type 0x0000

should never show up.

Actual results

ipv6: unknown message type 0x0000

shows up with some regularity

Versions

bdd2d52 on a somewhat recent Arch as of writing this issue.

Operating System Environment
-----------------------------
       Operating System: "Arch Linux" 
                 Kernel: Linux 4.20.3-arch1-1-ARCH x86_64 unknown

Installed compiler toolchains
-----------------------------
             native gcc: gcc (GCC) 8.2.1 20181127
      arm-none-eabi-gcc: arm-none-eabi-gcc (Arch Repository) 8.2.0
                avr-gcc: avr-gcc (GCC) 8.2.0
       mips-mti-elf-gcc: missing
             msp430-gcc: missing
   riscv-none-embed-gcc: missing
   xtensa-esp32-elf-gcc: missing
   xtensa-lx106-elf-gcc: missing
                  clang: clang version 7.0.1 (tags/RELEASE_701/final)

Installed compiler libs
-----------------------
   arm-none-eabi-newlib: "3.0.0"
    mips-mti-elf-newlib: missing
riscv-none-embed-newlib: missing
xtensa-esp32-elf-newlib: missing
xtensa-lx106-elf-newlib: missing
               avr-libc: "2.0.0" ("20150208")

Installed development tools
---------------------------
                  cmake: cmake version 3.13.3
               cppcheck: missing
                doxygen: 1.8.15
                 flake8: 3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 3.7.2 on Linux
                    git: git version 2.20.1
                   make: GNU Make 4.2.1
                openocd: missing
                 python: Python 3.7.2
                python2: Python 2.7.15
                python3: Python 3.7.2
             coccinelle: missing

[kaspar030]

I'm trying to reproduce. I got the extra case you've added once within 10 minutes.

Can you re-run your test with this: https://gist.github.com/1b447978ac89cfc77d6be51911e8b305
It adds printout of the msg_receive() return value, and changes msg_receive() to return different values depending on which case was hit.

[kaspar030]

Sorry, please use this: https://gist.github.com/92fa81d1ae60d5f134806027829ea736

[kaspar030]

It adds printout of the msg_receive() return value, and changes msg_receive() to return different values depending on which case was hit.

I only got three hits, they were all with res==1 so far.

[kaspar030]

@miri64 all the messages are supposed to be received from another thread, right? (not from isr?)

[miri64]

Hi @kaspar030 I was planning to work on an isolated test case today. Let me check your patches first.

@miri64 all the messages are supposed to be received from another thread, right? (not from isr?)

At least in the scenario, most should be from the thread itself (containing ICMPv6 echo replies or sent NDP messages), from the netif thread (ICMPv6 echo requests and received NDP messages), but also from the ISR (evtimer timer messages for NDP events).

[kaspar030]

I've added an to msg_receive() (at the top, write something to m->content.value, below in the failing case, read back and assert it has been changed). That triggers as @miri64's default case.

I've then tried to enable debug in msg.c, and it still triggers. It looks like a thread sitting in "msg_receive()" is being woken up before anyone has copied a message, which should not be possible.

This must be something timing sensitive, as enabling more debug output either postpones or "fixes" the problem.

I'm investigating...

[miri64]

I've added an to msg_receive() (at the top, write something to m->content.value, below in the failing case, read back and assert it has been changed). That triggers as @miri64's default case.

Can you rephrase / fix this sentence (or provide a patch)? I don't know what you mean.

[kaspar030]

Can you rephrase / fix this sentence (or provide a patch)? I don't know what you mean.

Sorry, patch is here. What I mean is that I've tried your check (memset, then see if type=0) directly into msg_receive(), as seen in the patch. As expected, it also triggers.