arp collector error on node with infiniband

[rbonser]

We have a cluster of hosts with infiniband cards that had an older version (1.1.2) of the node_exporter installed and running without error. We just updated to the latest version (1.6.1) and we are now seeing errors from the arp collector at each scrape.

Host operating system: output of uname -a

Linux xxxxx 4.18.0-425.19.2.el8_7.x86_64 #1 SMP Fri Mar 17 01:52:38 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b776)
build user: root@586879db11e5
build date: 20230717-12:10:52
go version: go1.20.6
platform: linux/amd64
tags: netgo osusergo static_build

node_exporter command line flags

/usr/local/bin/node_exporter --collector.systemd --collector.textfile --collector.textfile.directory=/var/lib/node_exporter --web.listen-address=0.0.0.0:9100 --web.telemetry-path=/metrics

Are you running node_exporter in Docker?

No

node_exporter log output

Aug 16 11:53:54 xxxxx node_exporter[256428]: ts=2023-08-16T15:53:54.696Z caller=collector.go:169 level=error msg="collector failed" name=arp duration_seconds=0.000139986 err="could not get ARP entries: failed to parse ARP entry: address 00:00:00:02:fe:80:00:00:00: invalid MAC address"

What did you do that produced an error?

upgraded the node_exporter

What did you expect to see?

no issues

What did you see instead?

error from the arp collector

[dswarbrick]

Prior to v1.6.0, the arp collector implemented the parsing in node_exporter. From v1.6.0 onwards, it uses github.com/prometheus/procfs to parse the entries (see #2551 and prometheus/procfs#545).

The previous implementation in node_exporter used a very naïve parser that only required each line to have 6 or more columns, and incremented a per-device counter without attempting to actually parse the link-layer addresses.

The procfs variant uses a much more comprehensive parser which uses net.ParseMAC() to parse the link-layer address into an ARPEntry struct, which indeed fails on values such as 00:00:00:02:fe:80:00:00:00.

[dswarbrick]

@rbonser Can you paste the first few lines of your /proc/net/arp (ensuring that some IPoIB entries are included)?

The Go docs state that net.ParseMAC() supports parsing IEEE 802 MAC-48, EUI-48, EUI-64 and 20-byte InfiniBand link-layer addresses. The value that you posted seems to be some weird 9-byte address.

[rbonser]

IP address       HW type     Flags       HW address            Mask     Device
10.255.48.90     0x20        0x2         00:00:08:ba:fe:80:00:00:00     *        ib0
10.255.48.99     0x20        0x2         00:00:06:7b:fe:80:00:00:00     *        ib0
10.255.48.1      0x20        0x2         00:00:00:02:fe:80:00:00:00     *        ib0
10.255.49.215    0x20        0x2         00:00:00:02:fe:80:00:00:00     *        ib0
10.255.48.104    0x20        0x2         00:00:04:9d:fe:80:00:00:00     *        ib0

[rbonser]

those addresses are truncated. for instance the mac address on 10.255.48.90 from the ip address output is 00:00:08:ba:fe:80:00:00:00:00:00:00:0c:42:a1:03:00:8e:e0:fa

[dswarbrick]

those addresses are truncated.

Yes - see my comment prometheus/procfs#545 (comment)

If the arp collector attempts to use net.ParseMAC() to parse the link-layer addresses read from /proc/net/arp, it is going to be fundamentally incompatible with IPoIB systems. The previous collector implementation only avoided this fate because it made no attempt to parse the link-layer addresses.

I feel that this is probably another case of procfs not being the best source of this information anymore, and rtnetlink would probably be a better way of getting the information (at least on Linux).

[srstsavage]

Yes, I'm seeing the same. Thank you dswarbrick for putting together a fix, hopefully it gets merged soon!

[mmospanenko]

having the same issue, maybe because I have two interfaces on the same mac, I don't know

[dswarbrick]

@mmospanenko I do not see any InfiniBand interfaces in your /proc/net/arp. This issue is specifically about InfiniBand, due to its link-layer addresses appearing truncated in /proc/net/arp.