node_exporter endpoint timeout, couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe #2852
Description
node exporter endpoint timeout on my 2 machines, and their log shows systemd collector fail to write unix file.
After restart, noded exporter is able to resume normal operation, but endpoint timeout would reappears a few hours later.
Host operating system: output of uname -a
Linux ydcs-vmziksyzjb 5.4.224-1.el7.elrepo.x86_64 #1 SMP Tue Nov 8 17:24:56 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version: output of node_exporter --version
node_exporter, version 1.4.1 (branch: HEAD, revision: a954c9f)
build user: root@0cb1bc5fbc4c
build date: 20221129-18:45:59
go version: go1.19.3
platform: linux/amd64
node_exporter command line flags
/usr/local/node_exporter/node_exporter/node_exporter --web.listen-address=:19101 --collector.systemd --web.config=/usr/local/node_exporter/node_exporter/node_exporter.yml --collector.textfile.directory=/usr/local/node_exporter/node_exporter/textfile.collected
other information: systemd version
systemd 243 (v243-31.ctl2)
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy
last reboot:
reboot system boot 5.4.224-1.el7.el Mon May 15 19:17 still running
reboot system boot 5.4.224-1.el7.el Thu May 11 09:42 still running
reboot system boot 5.4.224-1.el7.el Thu Mar 30 15:54 still running
reboot system boot 5.4.224-1.el7.el Thu Mar 30 11:48 - 15:54 (04:06)
reboot system boot 5.4.224-1.el7.el Mon Mar 6 10:14 - 11:48 (24+01:33)
node_exporter log output
Nov 19 12:48:02 ydcs-vmziksyzjb systemd[1]: Started node_exporter.
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.4.1, branch=HEAD, revision=a954c9fea6bcccea7f68a77a002a2a878ef1d60c)"
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.19.3, user=root@0cb1bc5fbc4c, date=20221129-18:45:59)"
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.989Z caller=node_exporter.go:185 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=systemd_linux.go:152 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=.+
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=systemd_linux.go:154 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+.(automount|device|mount|scope|slice)
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|v>
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.990Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|>
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=arp
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=bcache
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=bonding
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=btrfs
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=conntrack
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=cpu
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=cpufreq
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=diskstats
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=dmi
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=edac
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=entropy
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=fibrechannel
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=filefd
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=filesystem
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=hwmon
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=infiniband
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=ipvs
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=loadavg
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=mdadm
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=meminfo
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netclass
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netdev
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=netstat
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nfs
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nfsd
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=nvme
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=os
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=powersupplyclass
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=pressure
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=rapl
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=schedstat
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=selinux
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=sockstat
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=softnet
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=stat
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=systemd
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=tapestats
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=textfile
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=thermal_zone
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=time
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=timex
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=udp_queues
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=uname
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=vmstat
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=xfs
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:115 level=info collector=zfs
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=node_exporter.go:199 level=info msg="Listening on" address=:19101
Nov 19 12:48:02 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T04:48:02.991Z caller=tls_config.go:231 level=info msg="TLS is disabled." http2=false
Nov 20 00:05:43 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T16:05:43.723Z caller=collector.go:169 level=error msg="collector failed" name=systemd duration_seconds=240.056944888 err="couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe"
Nov 20 00:07:43 ydcs-vmziksyzjb node_exporter[1774175]: ts=2023-11-19T16:07:43.769Z caller=collector.go:169 level=error msg="collector failed" name=systemd duration_seconds=240.071481612 err="couldn't get dbus connection: write unix @->/run/systemd/private: write: broken pipe"
Are you running node_exporter in Docker?
no
What did you do that produced an error?
restart node_exporter, running serval hours, this problem occur every time.