Failed to inject filter ebpf for kprobe_skb_2: register r8 used twice

[kdrag0n]

With the static arm64 build of pwru v1.0.1, trying to use a filter results in the following error:

# ./pwru  'dst host 192.168.207.2'
2023/09/04 22:24:06 Failed to inject filter ebpf for kprobe_skb_2: register r8 used twice
# ./pwru  'port 80'
2023/09/04 22:26:28 Failed to inject filter ebpf for kprobe_skb_5: register r8 used twice

Kernel version: 6.4.14
Distro: Alpine edge

[brb]

Thanks for the issue. Does the same appear with v1.0.0?

[kdrag0n]

v1.0.0 seems to get past the filter generation, but fails while attaching kprobes because it doesn't have 5e6011d.

[brb]

@jschwinger233 Mind taking a look?

[kdrag0n]

Update: Just built v1.0.0 with the rlimit increased (as a workaround) and it seems to work fine.

[dakehero]

The pwru v1.0.1 amd64 build also has this issue on kernel 5.4

[brb]

Interesting, if I build v1.0.1 pwru from scratch (Archlinux, LLVM 16, gcc 13), I get it working. But with the released version (built by https://github.com/cilium/pwru/actions/runs/6071926301/job/16470942060, LLVM 13, gcc 12) I'm bumping into the same problem:

> sudo ./pwru 'port 80'
2023/09/05 09:49:58 Failed to inject filter ebpf for kprobe_skb_3: register r8 used twice

[jschwinger233]

Checking out upstream locally and running make local-release seems fine.

[jschwinger233]

I think it could be caused by wrong registers found for skb->data and skb->date_end.

In a good kprobemultipwru_bpfel_x86.o, we have

; 	bpf_printk("%d %d", data, data_end);
      79:	r1 = 16 ll
      81:	r2 = 6
      82:	r3 = r9
      83:	r4 = r8
      84:	call 6

In a bad one, we have

      76:	r3 = r8
      77:	r3 += r6
; 	void *data_end = skb_head + l4_off + len;
      78:	r8 += r7
; 	bpf_printk("%d %d", data, data_end);
      79:	r1 = 16 ll
      81:	r2 = 6
      82:	r4 = r8
      83:	call 6

In the pwru, the searching for registers is implemented inflexibly:

pwru/internal/libpcap/inject.go

Lines 126 to 145 in 80d7973

	var (
	dataReg asm.Register = 255
	dataEndReg asm.Register = 255
	)
	for idx := injectIdx - 1; idx >= 0; idx-- {
	inst := program.Instructions[idx]
	if inst.OpCode.ALUOp() == asm.Mov {
	if inst.Dst == asm.R3 {
	dataReg = inst.Src
	} else if inst.Dst == asm.R4 {
	dataEndReg = inst.Src
	}
	}
	if dataReg != 255 && dataEndReg != 255 {
	break
	}
	}
	if dataReg == 255 || dataEndReg == 255 {
	return errors.New("Cannot find the data / data_end registers")
	}

So pwru found r8 as both skb->data and skb->data_end, which explains error register r8 used twice