Kernel Address Off-by-one Error ksym resolution

[holmanb]

I am experiencing an issue involving the address returned from a kprobe being off by one from the value reported in /proc/kallsyms. I'm assuming that this is either a gobpf issue or user error, since some bcc tools that depend on symbol lookup appear to work on this system.

Kernel: 5.10.10-200.fc33.x86_64
libbpf: libbpf-0.1.0-1.fc33.x86_64

I've distilled a small example to demonstrates the issue:

package main
import (
        "os"
        "fmt"
        "time"
        bpf "github.com/iovisor/gobpf/bcc"
)

import "C"

const source string = `
#include <uapi/linux/ptrace.h>
struct key_t {
    u64 ip;
};
BPF_HASH(counts, struct key_t);
int do_count(struct pt_regs *ctx) {
    struct key_t key = {};
    u64 ip;
    key.ip = PT_REGS_IP(ctx);
    counts.increment(key); // update counter
    return 0;
}
`
type key_t struct {
    ip uint64
}

func main(){
        symbol := "mark_page_accessed"
        m := bpf.NewModule(source, []string{})
        defer m.Close()

        do_count_probe, err := m.LoadKprobe("do_count")
        if err != nil {
                fmt.Fprintf(os.Stderr, "Failed to load do_count : %s\n", err)
                os.Exit(1)
        }

        table := bpf.NewTable(m.TableId("counts"), m)
        err = m.AttachKprobe(symbol, do_count_probe, -1)
        if err != nil {
                fmt.Fprintf(os.Stderr, "Failed to attach mark_buffer_dirty: %s\n", err)
                os.Exit(1)
        }

        for it := table.Iter();;time.Sleep(time.Second) {
                for ;it.Next();{
                        kStr, err  := table.KeyBytesToStr(it.Key())
                        if err != nil {
                                panic(err)
                        }
                        fmt.Printf("address of %s is %s\n",symbol, kStr)
                        return
                 }
                 it = table.Iter()
        }
}

output:

address of mark_page_accessed is { 0xffffffff9626d761 }

which doesn't line up with the address exported via /proc/kallsyms

$ sudo grep " mark_page_accessed" /proc/kallsyms
ffffffff9626d760 T mark_page_accessed

I've seen this behavior for multiple symbols, so it's consistently off by one. I've demonstrated that I can do something like the following line to work around this issue temporarily for symbol lookups with gobpf/pkg/ksym, but this is ugly temporary workaround that I assume isn't reliable.

strval := strconv.FormatUint(bpf.GetHostByteOrder().Uint64(it.Key())-1,16)

[holmanb]

I was able to reproduce this on a different machine/os/kernel/libbpf version (using the example code above).

Kernel: 5.10.16-arch1-1
libbpf: 0.3-1

This is a pattern that is used extensively in bcc/tools.

Any ideas? I can provide additional debugging details if required.

[holmanb]

It looks like the off-by-one behavior is due to PT_REGS_IP() and occurs in iovisor/bcc as well.

However, iovisor/bcc has code to allow address lookup without the manual address manipulation workaround that I detailed above - bcc.BPF.ksym() is capable of resolving symbols from addresses given by PT_REGS_IP. Is there a reason that gobpf should diverge from bcc's behavior on this?

I'm tempted to dig through their ksym implementation to port it to gobpf. I've never contributed to this project, but I'd be willing to attempt that if that effort would be welcomed.