could not set ns(2907): Invalid argument

[withinboredom]

Logs say the following:

Nov 19 20:43:58 delta bpftune[950]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in non-global ns. Need to increase buffer size(s) to maximize throughput
Nov 19 20:43:58 delta bpftune[950]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (4096 131072 12288000) -> (4096 131072 15360000)
Nov 19 20:43:58 delta bpftune[950]: could not set ns(2907): Invalid argument
Nov 19 20:43:58 delta bpftune[950]: could not set ns(2907): Invalid argument

uname -a
Linux delta 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov  4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

There's thousands of these log entries. Should it realize this feature is unsupported and give up at some point?

[alan-maguire]

it looks like bpftune is trying to enter a netns and failing; we can look for such failures and mark the netns as gone. however I'd be interested to understand why this is failing. looks like the man page describes a number of reasons we might get back an EINVAL. Would you be able to run bpftune with debugging enabled (disable the service and run "bpftune -ds") so we can figure out exactly why it fails? in the meantime i'll push a fix for the simpler issue of noticing such failures and giving up. thanks!

[withinboredom]

Would you be able to run bpftune with debugging enabled (disable the service and run "bpftune -ds") so we can figure out exactly why it fails? in the meantime i'll push a fix for the simpler issue of noticing such failures and giving up. thanks!

Done. Here's a snippet of the logs around the error:

bpftune:  updated stat for tunable net.ipv4.tcp_rmem, scenario 0: 124
bpftune: event scenario [0] for tuner tcp_conn[5] netns 0 (global netns)
bpftune: tcp_conn: ::ffff:10.0.2.142: cong alg 'cubic': got rate_delivered 10, rtt 130, metric 1145097
bpftune: event scenario [0] for tuner tcp_conn[5] netns 0 (global netns)
bpftune: tcp_conn: ::ffff:10.0.0.67: cong alg 'bbr': got rate_delivered 5, rtt 315, metric 400000
bpftune: event scenario [0] for tuner tcp_conn[5] netns 0 (global netns)
bpftune: tcp_conn: ::ffff:10.0.2.153: cong alg 'bbr': got rate_delivered 10, rtt 130, metric 566265
bpftune: event scenario [0] for tuner tcp_buffer[6] netns 24578 (non-global netns)
bpftune: no tuner netns found for tuner 6, cookie 24578
bpftune: no tuner netns found for tuner 6, cookie 24578
bpftune: set caps (count 2)
bpftune: checking nsfs mnt /run/snapd/ns/canonical-livepatch.mnt
bpftune: set caps (count 3)
bpftune: could not set ns(225): Invalid argument
bpftune: drop caps (count 2)
bpftune: setns failed for for fd 225
bpftune: checking nsfs mnt /run/netns/cni-79c07cb9-c754-5873-e7e1-a6c80c49c9f4
bpftune: set caps (count 3)
bpftune: drop caps (count 2)
bpftune: set caps (count 3)
bpftune: drop caps (count 2)
bpftune: found netns fd 225 for cookie 2 via mnt /run/netns/cni-79c07cb9-c754-5873-e7e1-a6c80c49c9f4
bpftune: drop caps (count 1)
bpftune: writing sysctl '/proc/sys/net/ipv4/tcp_rmem' for netns_fd 225
bpftune: set caps (count 2)
bpftune: set caps (count 3)
bpftune: drop caps (count 2)
bpftune: set caps (count 3)
bpftune: Read net.ipv4.tcp_rmem[0] = 4096
bpftune: Read net.ipv4.tcp_rmem[1] = 131072
bpftune: Read net.ipv4.tcp_rmem[2] = 12288000
bpftune: drop caps (count 2)
bpftune: set caps (count 3)
bpftune: drop caps (count 2)
bpftune: drop caps (count 1)

[withinboredom]

Full logs can be provided as well, if that will help, they are just too big to fit here.

[alan-maguire]

thanks for the quick response! from the above the netns that appears to be causing the problems is this:

/run/snapd/ns/canonical-livepatch.mnt

so what happens is this - we get a message from our bpf program to update a tunable in a net namespace with the netns cookie associated with that netns. we then need to search through the net namespaces on the system for that namespace (if we don't already have netns state associated with it). it is in that search we get the error above for that netns. The solution is to only dump errors in the case that we are entering the network namespace for a more signifcant purpose - i.e. modifying a parameter - since otherwise everytime we iter over network namespaces we get a log error for any namespace we cannot enter as you see here.

if you're in a position to re-test, I just pushed #105 to the main branch and I think that should resolve this issue.

[withinboredom]

I pulled the latest main and ran make && make install but I'm still seeing the same output and debug logs look the same as before. I'm not certain how to check that it is actually the commit it should be. (the --version is the same output as before), so maybe I did something wrong, but I double-checked that 870d4662a4496e98585d5fc22815d636682faf6f is the commit I checked out.

[withinboredom]

Oh, yep. #105 appears to have fixed it (I needed to run make clean before make)!

[alan-maguire]

great, thanks for testing!