many_netns_test.sh fails test

[rknobbe]

running the test suite catches an error on many_netns_test.sh|wmem

TEST_ID=$PPID  bash many_netns_test.sh
many_netns_test.sh|wmem test to 192.168.168.1:5201 ipv4 opts  |START
net.ipv4.tcp_wmem = 4096 16384 16384
net.ipv4.tcp_wmem = 4096 16384 16384
Running baseline...
Running test...
net.ipv4.tcp_wmem = 4096 16384 4194304
wmem before 16384 ; after 40389085
netns wmem before 16384 ; after 16384
net.ipv6.conf.all.disable_ipv6 = 0
Output of last command:

Connecting to host 192.168.168.1, port 5201
[  5] local 192.168.168.2 port 57398 connected to 192.168.168.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.68 GBytes  22997 Mbits/sec    0   6.96 MBytes       
[  5]   1.00-2.00   sec  4.13 GBytes  35473 Mbits/sec    0   6.96 MBytes       
[  5]   2.00-3.00   sec  3.60 GBytes  30927 Mbits/sec    0   6.96 MBytes       
[  5]   3.00-4.00   sec  3.84 GBytes  32996 Mbits/sec    0   6.96 MBytes       
[  5]   4.00-5.00   sec  3.71 GBytes  31908 Mbits/sec    0   6.96 MBytes       o sadm sds apst 
[  5]   5.00-6.00   sec  3.79 GBytes  32583 Mbits/sec    0   6.96 MBytes       
[  5]   6.00-7.00   sec  3.73 GBytes  32061 Mbits/sec    0   6.96 MBytes       
[  5]   7.00-8.00   sec  3.83 GBytes  32923 Mbits/sec    0   6.96 MBytes       
[  5]   8.00-9.00   sec  3.72 GBytes  31948 Mbits/sec    0   6.96 MBytes       
[  5]   9.00-10.00  sec  3.81 GBytes  32676 Mbits/sec    0   6.96 MBytes       0
- - - - - - - - - - - - - - - - - - - - - - - - -0
[ ID] Interval           Transfer     Bitrate         Retr0
[  5]   0.00-10.00  sec  37.0 GBytes  31780 Mbits/sec    0             sender0
[  5]   0.00-10.00  sec  37.0 GBytes  31780 Mbits/sec                  receiver

iperf Done.
many_netns_test.sh|wmem test to 192.168.168.1:5201 ipv4 opts  |FAIL; error 1|
make[1]: *** [Makefile:85: many_netns_test] Error 1
make[1]: Leaving directory '/home/roger/bpftune/test'
make: *** [Makefile:49: test] Error 2
roger@m910q:~/bpftune$ uname -a
Linux m910q 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_
64 x86_64 x86_64 GNU/Linux
roger@m910q:~/bpftune$ lsb_release -a
No LSB modules are available.
Distributor ID:Ubuntu
Description:Ubuntu 24.04.1 LTSout
Release:24.04
Codename:noble

roger@m910q:~/bpftune$ git show
commit efe78a0cf85d1b4804ff1c14bcaa89684b7181e9 (HEAD -> main, origin/main, origin/HEAD)
Merge: 870d466 aaec117
Author: Alan Maguire <32452915+alan-maguire@users.noreply.github.com>
Date:   Thu Nov 21 12:47:00 2024 +0000

    Merge pull request #106 from oracle/fdleak
    
    Fdleak

[alan-maguire]

thanks for the report! looks like the namespace wmem was not updated as expected; i'll investigate further. as a matter of interest

• is this a consistent or intermittent failure?
• if consistent, does the wmem test fail too? (bash wmem_test.sh)?
  thanks!

[rknobbe]

Yes, happens every time I run it. Same with wmem_test.sh and rmem_test.sh

roger@m910q:~/bpftune/test$ sudo bash ./wmem_test.sh
[sudo] password for roger:
./wmem_test.sh|wmem test to 192.168.168.1:5201 ipv4 opts  |START
net.ipv4.tcp_wmem = 4096 16384 16384
net.ipv4.tcp_wmem = 4096 16384 16384
Running baseline...
Running test...
net.ipv4.tcp_wmem = 4096 16384 4194304
wmem before 16384 ; after 50486356
netns wmem before 16384 ; after 16384
net.ipv6.conf.all.disable_ipv6 = 0
Output of last command:

Connecting to host 192.168.168.1, port 5201
[  5] local 192.168.168.2 port 46122 connected to 192.168.168.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.67 GBytes  31492 Mbits/sec    0   6.18 MBytes
[  5]   1.00-2.00   sec  4.31 GBytes  37050 Mbits/sec    0   6.81 MBytes
[  5]   2.00-3.00   sec  4.17 GBytes  35844 Mbits/sec    0   6.81 MBytes
[  5]   3.00-4.00   sec  4.38 GBytes  37586 Mbits/sec    0   6.81 MBytes
[  5]   4.00-5.00   sec  3.97 GBytes  34096 Mbits/sec    0   6.81 MBytes
[  5]   5.00-6.00   sec  3.68 GBytes  31606 Mbits/sec    0   6.81 MBytes
[  5]   6.00-7.00   sec  3.92 GBytes  33666 Mbits/sec    0   6.81 MBytes
[  5]   7.00-8.00   sec  3.77 GBytes  32397 Mbits/sec    0   6.81 MBytes
[  5]   8.00-9.00   sec  3.75 GBytes  32167 Mbits/sec    0   6.81 MBytes
[  5]   9.00-10.00  sec  3.82 GBytes  32795 Mbits/sec    0   6.81 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  39.5 GBytes  33920 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  39.5 GBytes  33920 Mbits/sec                  receiver

iperf Done.
./wmem_test.sh|wmem test to 192.168.168.1:5201 ipv4 opts  |FAIL; error 1|
roger@m910q:~/bpftune/test$
roger@m910q:~/bpftune/test$ sudo bash ./rmem_test.sh
./rmem_test.sh|rmem test to 192.168.168.1:5201 ipv4 opts  |START
net.ipv4.tcp_rmem = 4096 131072 131072
net.ipv4.tcp_rmem = 4096 131072 131072
Running baseline...
Running test...
net.ipv4.tcp_rmem = 4096 131072 7864320
rmem before 131072 ; after 256000
netns rmem before 131072 ; after 131072
net.ipv6.conf.all.disable_ipv6 = 0
Output of last command:

Connecting to host 192.168.168.1, port 5201
[  5] local 192.168.168.2 port 39162 connected to 192.168.168.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.53 GBytes  21749 Mbits/sec    0    175 KBytes
[  5]   1.00-2.00   sec  2.33 GBytes  19996 Mbits/sec    0    175 KBytes
[  5]   2.00-3.00   sec  2.48 GBytes  21284 Mbits/sec    0    175 KBytes
[  5]   3.00-4.00   sec  2.53 GBytes  21716 Mbits/sec    0    175 KBytes
[  5]   4.00-5.00   sec  2.57 GBytes  22046 Mbits/sec    0    175 KBytes
[  5]   5.00-6.00   sec  2.60 GBytes  22356 Mbits/sec    0    175 KBytes
[  5]   6.00-7.00   sec  2.60 GBytes  22294 Mbits/sec    0    175 KBytes
[  5]   7.00-8.00   sec  2.58 GBytes  22149 Mbits/sec    0    175 KBytes
[  5]   8.00-9.00   sec  2.61 GBytes  22432 Mbits/sec    0    175 KBytes
[  5]   9.00-10.00  sec  2.50 GBytes  21451 Mbits/sec    0    175 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  25.3 GBytes  21761 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  25.3 GBytes  21760 Mbits/sec                  receiver

iperf Done.
./rmem_test.sh|rmem test to 192.168.168.1:5201 ipv4 opts  |FAIL; error 1|
roger@m910q:~/bpftune/test$

[alan-maguire]

what does "bpftune -S" report for the failing system?

[rknobbe]

bpftune: bpftune works fully
bpftune: bpftune supports per-netns policy (via netns cookie)

[rknobbe]

if it's helpful I'm also seeing these in dmesg while I'm running the tests.

[Fri Nov 22 11:37:52 2024] bpftune[850138]: segfault at 76996da39002 ip 000076996be06c32 sp 00007ffc4864ec30 error 4 in tcp_buffer_tuner.so[76996be02000+6000] likely on CPU 3 (core 3, socke
t 0)
[Fri Nov 22 11:37:52 2024] Code: 74 b7 ff ff e9 4c 04 00 00 48 8b 85 28 ff ff ff 8b 40 40 83 f8 03 75 20 48 8b 85 28 ff ff ff 48 8b 40 48 48 8b 80 a8 00 00 00 <0f> b6 40 02 0f b6 c0 85 c0 0
f 95 c0 eb 4d 48 8b 85 28 ff ff ff 8b
[Fri Nov 22 11:38:27 2024] bpftune[850675]: segfault at 7724d7416002 ip 00007724d5806c61 sp 00007ffcd952ea00 error 4 in tcp_buffer_tuner.so[7724d5802000+6000] likely on CPU 2 (core 2, socke
t 0)
[Fri Nov 22 11:38:27 2024] Code: b6 c0 85 c0 0f 95 c0 eb 4d 48 8b 85 28 ff ff ff 8b 40 40 83 f8 02 75 20 48 8b 85 28 ff ff ff 48 8b 40 48 48 8b 80 a0 00 00 00 <0f> b6 40 02 0f b6 c0 85 c0 0
f 95 c0 eb 1e 48 8b 85 28 ff ff ff 48
[Fri Nov 22 11:39:03 2024] bpftune[851139]: segfault at 7b3f03b20002 ip 00007b3f02006c32 sp 00007ffce1411fe0 error 4 in tcp_buffer_tuner.so[7b3f02002000+6000] likely on CPU 1 (core 1, socke
t 0)
[Fri Nov 22 11:39:03 2024] Code: 74 b7 ff ff e9 4c 04 00 00 48 8b 85 28 ff ff ff 8b 40 40 83 f8 03 75 20 48 8b 85 28 ff ff ff 48 8b 40 48 48 8b 80 a8 00 00 00 <0f> b6 40 02 0f b6 c0 85 c0 0
f 95 c0 eb 4d 48 8b 85 28 ff ff ff 8b
[Fri Nov 22 11:39:37 2024] bpftune[851631]: segfault at 76b376846002 ip 000076b374c06c61 sp 00007fffa5023290 error 4 in tcp_buffer_tuner.so[76b374c02000+6000] likely on CPU 2 (core 2, socke
t 0)
[Fri Nov 22 11:39:37 2024] Code: b6 c0 85 c0 0f 95 c0 eb 4d 48 8b 85 28 ff ff ff 8b 40 40 83 f8 02 75 20 48 8b 85 28 ff ff ff 48 8b 40 48 48 8b 80 a0 00 00 00 <0f> b6 40 02 0f b6 c0 85 c0 0
f 95 c0 eb 1e 48 8b 85 28 ff ff ff 48

[alan-maguire]

ah, that's the problem, thanks! This may be either a genuine segmentation fault, or a segmentation fault resulting from dependency issues in the makefiles. I've seen segmentation faults caused by (I think) tuners not getting rebuilt when libbpftune is so I'd suggest trying "make clean; make; sudo make install" at the toplevel and see if the problem persists. if it does it would be great the get the stack trace of the associated core file. I'll investigate the makefile dependencies also

[rknobbe]

I took those steps and am still seeing the same issue. Do you have a pointer on how to get a stack trace?

…

On Mon, Nov 25, 2024 at 2:05 AM Alan Maguire ***@***.***> wrote: ah, that's the problem, thanks! This may be either a genuine segmentation fault, or a segmentation fault resulting from dependency issues in the makefiles. I've seen segmentation faults caused by (I think) tuners not getting rebuilt when libbpftune is so I'd suggest trying "make clean; make; sudo make install" at the toplevel and see if the problem persists. if it does it would be great the get the stack trace of the associated core file. I'll investigate the makefile dependencies also — Reply to this email directly, view it on GitHub <#107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGD5GN4T7ROZ4FAQBLWUGT2CLY7BAVCNFSM6AAAAABSH7GSBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJXGQ4TQMRXHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alan-maguire]

If you have a core dump the easiest way is via gdb ; running "gdb bpftune /path/to/corefile" and then "bt" At the gdb prompt should do it

[rknobbe]

Here's the trace:

Reading symbols from /usr/sbin/bpftune...

warning: Can't open file anon_inode:bpf-map which was expanded to anon_inode:bpf-map during file-backed mapping note processing

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 57234]
[New LWP 57239]
[New LWP 57240]

warning: could not find '.gnu_debugaltlink' file for /lib/x86_64-linux-gnu/libcap.so.2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/bpftune -c /var/run/bpftune/cgroupv2'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000073436fa06c32 in event_handler (tuner=0x598cd68bdd80, event=0x734370d95c08, ctx=0x0) at tcp_buffer_tuner.c:209
209             near_memory_exhaustion = bpftuner_bpf_var_get(tcp_buffer, tuner,
[Current thread is 1 (Thread 0x734371463b80 (LWP 57234))]
(gdb) bt
#0  0x000073436fa06c32 in event_handler (tuner=0x598cd68bdd80, event=0x734370d95c08, ctx=0x0) at tcp_buffer_tuner.c:209
#1  0x0000734371206686 in bpftune_ringbuf_event_read (ctx=0x0, data=0x734370d95c08, size=248) at libbpftune.c:786
#2  0x00007343714df12e in ?? () from /lib/x86_64-linux-gnu/libbpf.so.1
#3  0x00007343714e1fc9 in ring_buffer.poll () from /lib/x86_64-linux-gnu/libbpf.so.1
#4  0x0000734371206773 in bpftune_ring_buffer_poll (ring_buffer=0x598cd691d8d0, interval=100) at libbpftune.c:824
#5  0x0000598cd6535699 in main (argc=3, argv=0x7fffe1d35078) at bpftune.c:420

[alan-maguire]

thanks! looks like the segmentation fault occurs when dereferencing BPF program variable tuner->skel->near_memory_pressure . it's odd that tuner->skel or tuner->skel->value would be NULL. Would you mind running the following commands in gdb for this core dump? (i'm afraid I can't reproduce this at my end at all)

print *(struct bpftuner *)0x598cd68bdd80

that should show us the state of the tuner when this crash happens. thanks again!

[rknobbe]

(gdb) bt #0 0x000073436fa06c32 in event_handler (tuner=0x598cd68bdd80, event=0x734370d95c08, ctx=0x0) at tcp_buffer_tuner.c:209 #1 0x0000734371206686 in bpftune_ringbuf_event_read (ctx=0x0, data=0x734370d95c08, size=248) at libbpftune.c:786 #2 0x00007343714df12e in ?? () from /lib/x86_64-linux-gnu/libbpf.so.1 #3 0x00007343714e1fc9 in ring_buffer.poll () from /lib/x86_64-linux-gnu/libbpf.so.1 #4 0x0000734371206773 in bpftune_ring_buffer_poll (ring_buffer=0x598cd691d8d0, interval=100) at libbpftune.c:824 #5 0x0000598cd6535699 in main (argc=3, argv=0x7fffe1d35078) at bpftune.c:420 (gdb) print *(struct bpftuner *)0x598cd68bdd80 $1 = {id = 4, state = BPFTUNE_MANUAL, netns = {next = 0x598cd691ee00, netns_cookie = 0, state = BPFTUNE_INACTIVE}, path = 0x0, handle = 0x598cd68bcfe0, name = 0x73436fa084dc "tcp_buffer", skeleton = 0x598cd68bdef0, bpf_support = BPFTUNE_SUPPORT_NORMAL, rollback = false, skel = 0x598cd68d01c0, obj = 0x598cd68d0320, init = 0x73436fa03b5f <init>, fini = 0x73436fa06aa3 <fini>, strategies = 0x0, strategy = 0x0, ring_buffer_map = 0x598cd68f1b70, ring_buffer_map_fd = 50, corr_map = 0x0, corr_map_fd = 0, netns_map = 0x598cd68f1c10, netns_map_fd = 17, event_handler = 0x73436fa06ae6 <event_handler>, num_tunables = 4, tunables = 0x598cd68f37f0, num_scenarios = 6, scenarios = 0x73436fc9e1c0 <scenarios>} (gdb)

…

On Mon, Nov 25, 2024 at 9:59 AM Alan Maguire ***@***.***> wrote: thanks! looks like the segmentation fault occurs when dereferencing BPF program variable tuner->skel->near_memory_pressure . it's odd that tuner->skel or tuner->skel->value would be NULL. Would you mind running the following commands in gdb for this core dump? (i'm afraid I can't reproduce this at my end at all) print *(struct bpftuner *)0x598cd68bdd80 that should show us the state of the tuner when this crash happens. thanks again! — Reply to this email directly, view it on GitHub <#107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGD5GLXECDUZWDQPFH2WI32CNQR7AVCNFSM6AAAAABSH7GSBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJYGY4DSMZVGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alan-maguire]

thanks! this tells us that an inactive tuner got an event, triggering the segmentation fault. i've just pushed a fix to main which I think will address that, if you get a chance to retest with it that would be great! I'm reasonably hopeful it should resolve the segmentation fault, though I'm not sure if it will fix the test failures yet..

[rknobbe]

OK, thanks. Yes that fixed the segfault but I'm still getting a fail on rmem_test.sh (for example).

roger@m910q:~/bpftune/test$ sudo bash  ./rmem_test.sh
./rmem_test.sh|rmem test to 192.168.168.1:5201 ipv4 opts  |START
net.ipv4.tcp_rmem = 4096 131072 131072
net.ipv4.tcp_rmem = 4096 131072 131072
Running baseline...
Running test...
net.ipv4.tcp_rmem = 4096 131072 6291456
rmem before 131072 ; after 204800
netns rmem before 131072 ; after 131072
net.ipv6.conf.all.disable_ipv6 = 0
Output of last command:

Connecting to host 192.168.168.1, port 5201
[  5] local 192.168.168.2 port 34670 connected to 192.168.168.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.33 GBytes  20007 Mbits/sec    2    175 KBytes
[  5]   1.00-2.00   sec  2.65 GBytes  22722 Mbits/sec    0    175 KBytes
[  5]   2.00-3.00   sec  2.55 GBytes  21946 Mbits/sec    0    175 KBytes
[  5]   3.00-4.00   sec  2.63 GBytes  22564 Mbits/sec    0    175 KBytes
[  5]   4.00-5.00   sec  2.52 GBytes  21640 Mbits/sec    0    175 KBytes
[  5]   5.00-6.00   sec  2.68 GBytes  23017 Mbits/sec    0    175 KBytes
[  5]   6.00-7.00   sec  2.40 GBytes  20633 Mbits/sec    0    175 KBytes
[  5]   7.00-8.00   sec  2.11 GBytes  18114 Mbits/sec    0    175 KBytes
[  5]   8.00-9.00   sec  2.15 GBytes  18469 Mbits/sec    1    175 KBytes
[  5]   9.00-10.00  sec  2.15 GBytes  18459 Mbits/sec    0    175 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  24.2 GBytes  20761 Mbits/sec    3             sender
[  5]   0.00-10.00  sec  24.2 GBytes  20758 Mbits/sec                  receiver

iperf Done.
./rmem_test.sh|rmem test to 192.168.168.1:5201 ipv4 opts  |FAIL; error 1|

[alan-maguire]

thanks for the quick test! would you mind attaching the output of a verbose test run, i.e.

VERBOSE=1 sudo -E bash rmem_test.sh

[rknobbe]

verbose.txt