Unable to debug binaries that require locked memory

[camden-xage]

1. What version of Delve are you using (dlv version) - 1.21.0
2. What version of Go are you using? (go version)? - 1.21.0
3. What operating system and processor architecture are you using? - Ubuntu 22.04 x86_64
4. What did you do? - Ran delve on a Go binary that needs to lock memory (minimal example below)
5. What did you expect to see? - Successful debug (works when not run in delve)
6. What did you see instead?- The binary doesn't have the ability to lock memory, seemingly because delve changes the locked memory limit

I have this minimal example that calls Mlock (originally ran into this when debugging with memguard):

package main

import (
	"fmt"
	"os"
	"time"

	"golang.org/x/sys/unix"
)

func main() {
	fmt.Println("PID:", os.Getpid())

	mem := make([]byte, 32)
	err := unix.Mlock(mem)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	// Some time to read proc files when running normally
	time.Sleep(100 * time.Second)
}

If I run build and run this normally, it is able to properly lock memory as it inherits the normal locked memory limit. When running with delve, both delve and the go binary it runs have there limits set down to 0 which prevents the binary from being able to lock memory. Does delve manually alter its limits and, if so, is there a way around it?

Running the binary with go run main.go (no error):

❯ cat /proc/34929/limits | grep -i locked
Max locked memory         4166123520           4166123520           bytes     

Running the binary with dlv debug (will get operation not permitted because of the limit change). Both dlv and the debug binary it runs have these limits:

❯ cat /proc/35602/limits | grep -i locked
Max locked memory         0                    0                    bytes  

It does seem like delve is changing the limit, but I simply used strace on both runs and looked at the *limit syscalls.

Run with go:

❯ go build -o main main.go && strace -c ./main
PID: 39667
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         1           setrlimit
  ...
------ ----------- ----------- --------- --------- ------------------
100.00    0.000000           0       168           total

Run with dlv debug:

❯ strace -c dlv debug
Type 'help' for list of commands.
(dlv) exit
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
  0.01    0.000013           4         3         1 prlimit64
  0.01    0.000007           7         1           setrlimit
  0.00    0.000004           4         1           getrlimit
  ...
------ ----------- ----------- --------- --------- ------------------
100.00    0.136488         272       501        68 total

Is is accurate that delve manually lowers it's locked memory limit? Is there a way around that, or are binaries that lock memory not supported?

[camden-xage]

This issue from the memguard repository seems to be encountering the same issue. The panic afterwards is a side effect, but the error that's encountered is the same and also only happens when debugging. awnumar/memguard#150

[aarzilli]

Looks like it's coming from

delve/vendor/github.com/cilium/ebpf/rlimit/rlimit.go

Line 41 in 7fb9dda

zeroLimit := unix.Rlimit{Cur: 0, Max: oldLimit.Max}

[camden-xage]

Looks like it's coming from

delve/vendor/github.com/cilium/ebpf/rlimit/rlimit.go

Line 41 in 7fb9dda

zeroLimit := unix.Rlimit{Cur: 0, Max: oldLimit.Max}

Yeah, I just noticed that in the code as well. I guess that affects the memory limit even we aren't tracing with ebpf? That function is run in that package's init() so I'm guessing that the delve package that imports it is being imported during normal debugging and causing that init to be called.

delve/vendor/github.com/cilium/ebpf/rlimit/rlimit.go

Lines 23 to 36 in 7fb9dda

	func init() {
	// We have to run this feature test at init, since it relies on changing
	// RLIMIT_MEMLOCK. Doing so is not safe in a concurrent program. Instead,
	// we rely on the initialization order guaranteed by the Go runtime to
	// execute the test in a safe environment:
	//
	// the invocation of init functions happens in a single goroutine,
	// sequentially, one package at a time.
	//
	// This is also the reason why RemoveMemlock is in its own package:
	// we only want to run the initializer if RemoveMemlock is called
	// from somewhere.
	haveMemcgAccounting = detectMemcgAccounting()
	}

[aarzilli]

Can you try to run #3491 and see if it fixes your problem, I think this is a bug with cilium/ebpf.

[camden-xage]

Sweet, that seems to have worked. No more operation not permitted error and the limits seem correct now.

1.21.0:

❯ dlv debug
Type 'help' for list of commands.
(dlv) continue
PID: 75835
operation not permitted
Process 75835 has exited with status 1
(dlv) exit

❯ pgrep -P 75691
75835

# delve
❯ cat /proc/75691/limits | grep locked
Max locked memory         0                    0                    bytes     

# go binary
❯ cat /proc/75835/limits | grep locked
Max locked memory         0                    0                    bytes     

PR build

❯ dlv debug
Type 'help' for list of commands.
(dlv) continue
PID: 76693
Process 76693 has exited with status 0
(dlv) exit

❯ pgrep -P 76557
76693

# delve
❯ cat /proc/76557/limits | grep locked
Max locked memory         4166123520           4166123520           bytes     

# go binary
❯ cat /proc/76693/limits | grep locked
Max locked memory         4166123520           4166123520           bytes