Bug: NTP - chrony does not sync time over MGMT VRF

[gupurush]

Is it platform specific

generic

Importance or Severity

High

Description of the bug

When MGMT VRF is configured, chrony service fails, after restarting it, it fails to synchronize time with NTP servers. The service starts successfully but cannot establish communication with configured NTP servers, resulting in "Not synchronised" status and zero reach values.

Steps to Reproduce

Configure NTP servers
Configure MGMT VRF for NTP
Restart chrony service: systemctl restart chrony
Wait a few seconds and check NTP status: show ntp

Actual Behavior and Expected Behavior

Actual Behavior
Chrony service starts and runs in mgmt VRF context
show ntp initially returns 506 Cannot talk to daemon
After daemon starts, NTP does not sync time

Expected Behavior
Chrony service should start successfully in mgmt VRF
show ntp should show synchronized status within a few minutes

Relevant log output

root@chu-dut:/home/admin# show ntp
MGMT_VRF_CONFIG is not present.
Reference ID    : 4A86B6C4 (fdcc::1.55.15.11  )
Stratum         : 3
Ref time (UTC)  : Thu Oct 23 22:43:33 2025
System time     : 0.000010565 seconds slow of NTP time
Last offset     : -0.000007988 seconds
RMS offset      : 0.000010227 seconds
Frequency       : 16.315 ppm slow
Residual freq   : -0.002 ppm
Skew            : 0.019 ppm
Root delay      : 0.001095369 seconds
Root dispersion : 0.016145004 seconds
Update interval : 259.0 seconds
Leap status     : Normal
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 1.55.15.11                    2   8   377    50    -36us[  -44us] +/-   23ms
^* fdcc::1.55.15.11             2   8   377    48    -38us[  -46us] +/-   23ms

root@chu-dut:/home/admin# systemctl status chrony
× chrony.service - chrony, an NTP client/server
     Loaded: loaded (/lib/systemd/system/chrony.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/chrony.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Thu 2025-10-23 22:48:40 UTC; 1min 37s ago
   Duration: 3h 52min 14.537s
       Docs: man:chronyd(8)
             man:chronyc(1)
             man:chrony.conf(5)

Oct 23 22:48:38 chu-dut systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 23 22:48:39 chu-dut chronyd-starter.sh[214107]: Starting NTP server in mgmt-vrf
Oct 23 22:48:40 chu-dut chronyd-starter.sh[214107]: Invalid VRF name
Oct 23 22:48:40 chu-dut systemd[1]: chrony.service: Control process exited, code=exited, status=255/EXCEPTION
Oct 23 22:48:40 chu-dut systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 23 22:48:40 chu-dut systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.

root@chu-dut:/home/admin#
root@chu-dut:/home/admin# systemctl status chrony
● chrony.service - chrony, an NTP client/server
     Loaded: loaded (/lib/systemd/system/chrony.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/chrony.service.d
             └─override.conf
     Active: active (running) since Thu 2025-10-23 22:50:53 UTC; 6s ago
       Docs: man:chronyd(8)
             man:chronyc(1)
             man:chrony.conf(5)
    Process: 221323 ExecStartPre=/usr/bin/chrony-config.sh (code=exited, status=0/SUCCESS)
    Process: 221327 ExecStart=/usr/local/sbin/chronyd-starter.sh (code=exited, status=0/SUCCESS)
   Main PID: 221331 (chronyd)
      Tasks: 2 (limit: 37461)
     Memory: 1.7M
     CGroup: /system.slice/chrony.service
             └─vrf
               └─mgmt
                 ├─221331 /usr/sbin/chronyd -F 1
                 └─221332 /usr/sbin/chronyd -F 1

Oct 23 22:50:52 chu-dut systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 23 22:50:52 chu-dut chronyd-starter.sh[221327]: Starting NTP server in mgmt-vrf
Oct 23 22:50:53 chu-dut chronyd[221331]: chronyd version 4.3 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYNCDNS>
Oct 23 22:50:53 chu-dut chronyd[221331]: nss_tacplus: /etc/tacplus_nss.conf fopen failed
Oct 23 22:50:53 chu-dut chronyd[221331]: nss_tacplus: bad config or server line for nss_tacplus
Oct 23 22:50:53 chu-dut chronyd[221331]: Frequency -16.326 +/- 0.046 ppm read from /var/lib/chrony/chrony.drift
Oct 23 22:50:53 chu-dut chronyd[221331]: Using right/UTC timezone to obtain leap second data
Oct 23 22:50:53 chu-dut chronyd[221331]: Loaded seccomp filter (level 1)
Oct 23 22:50:53 chu-dut systemd[1]: Started chrony.service - chrony, an NTP client/server.

root@chu-dut:/home/admin# show ntp
506 Cannot talk to daemon

root@chu-dut:/home/admin# show ntp
Reference ID    : 00000000 ()
Stratum         : 0
Ref time (UTC)  : Thu Jan 01 00:00:00 1970
System time     : 0.000000000 seconds slow of NTP time
Last offset     : +0.000000000 seconds
RMS offset      : 0.000000000 seconds
Frequency       : 16.326 ppm slow
Residual freq   : +0.000 ppm
Skew            : 0.000 ppm
Root delay      : 1.000000000 seconds
Root dispersion : 1.000000000 seconds
Update interval : 0.0 seconds
Leap status     : Not synchronised
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
^? 1.55.15.11                    0   6     0     -     +0ns[   +0ns] +/-    0ns
^? fdcc::1.55.15.11             0   6     0     -     +0ns[   +0ns] +/-    0ns

Output of show version, show techsupport

Attach files (if any)

No response

[prabhataravind]

Please reach out to @saiarcot895 for review help.

[ben-twn1]

Hi @gupurush and @prabhataravind,

I encountered the same issue on branch 2025 and identified two possible root causes:

1. When configuring the mgmt VRF, the NTP|global entry in CONFIG_DB is not updated accordingly.
2. The bindacqdevice directive in chrony service depends on SO_BINDTODEVICE option, which requires root privileges. However, due to limited permissions in mgmt VRF, chronyd fails to bind to the interface.

admin@sonic:~$ sudo ip vrf exec mgmt strace -e setsockopt -p $(pgrep chronyd | head -1) 2>&1 | grep BINDTODEVICE
setsockopt(4, SOL_SOCKET, SO_BINDTODEVICE, "eth0\0", 5) = -1 EPERM (Operation not permitted)
setsockopt(4, SOL_SOCKET, SO_BINDTODEVICE, "eth0\0", 5) = -1 EPERM (Operation not permitted)
setsockopt(4, SOL_SOCKET, SO_BINDTODEVICE, "eth0\0", 5) = -1 EPERM (Operation not permitted)

For the first issue, I referred to #3456 and was able to resolve it successfully.

For the second issue, which seems related to the kernel setsockopt() function and the capabilities of chrony service, I’m not very familiar with this aspect. The following are some temporary workarounds that worked in our tests, in case they might be helpful:

• Remove the binding configuration (i.e., delete bindacqdevice eth0 from /etc/chrony/chrony.conf) and let chronyd discover its own routing.
• Change bindacqdevice eth0 to bindacqaddress <eth0_IP> in /etc/chrony/chrony.conf.
• Add -u root to the exec ip vrf exec CLI in chronyd-starter.sh, which will run chronyd as root instead of _chrony.

I hope the above information is helpful and look forward to any fundamental solutions or suggestions. Thanks!