Fix rsyslogd memory growth in syncd swss containers over long term by tirupatihemanth · Pull Request #25874 · sonic-net/sonic-buildimage

tirupatihemanth · 2026-03-04T04:05:31Z

Why I did it

We observed long-term rsyslogd memory growth in syncd container.
Deep diagnostics (impstats) showed imuxsock.ratelimit.numratelimiters growing continuously (about ~2/min), while queue depth stayed near zero, indicating sender/PID churn rather than queue backlog.
phcsync.sh runs every 60 seconds and repeatedly invokes phc_ctl for /dev/ptp* devices. These short-lived process invocations contribute to new sender identities seen by imuxsock, which correlates with ratelimiter-state growth and memory increase over time because of data structures stored by rsyslogd for ratelimiting.

Work item tracking

Microsoft ADO (number only):

How I did it

Updated phcsync.sh in SONiC to keep successful phc_ctl execution silent:
Use phc_ctl -q -Q ... >/dev/null 2>&1
Keep explicit error handling and error logs on non-zero exit.
Added stable logger identity in service debug helpers:
logger -i "$$" -- "$1" in syncd_common.sh and swss.sh. This reduces per-call sender churn during script execution phases (start/wait/stop).

syncd
Every second we currently see following log from syncd and it creates a new ratelimiter context in rsyslogd because of new PID each time

syslog.1:15477:2026 Mar  2 22:25:01.754471 sonic NOTICE syncd#phc_ctl: [561375.455] set clock time to 1772490301.754287973 or Mon Mar  2 22:25:01 2026

logger commands
before

Mar 04 03:55:44 sonic root[1775781]: Starting swss service...
Mar 04 03:55:44 sonic root[1775785]: Locking /tmp/swss-syncd-lock from swss service
Mar 04 03:55:44 sonic root[1775792]: Locked /tmp/swss-syncd-lock (10) from swss service
Mar 04 03:55:44 sonic root[1775816]: Warm boot flag: swss false.
Mar 04 03:55:44 sonic root[1775822]: Flushing APP, ASIC, COUNTER, CONFIG, and partial STATE databases ...
Mar 04 03:55:45 sonic root[1776045]: Started swss service...
Mar 04 03:55:45 sonic root[1776051]: Unlocking /tmp/swss-syncd-lock (10) from swss service

After

Mar 04 03:58:52 sonic root[1891651]: Starting swss service...
Mar 04 03:58:52 sonic root[1891651]: Locking /tmp/swss-syncd-lock from swss service
Mar 04 03:58:52 sonic root[1891651]: Locked /tmp/swss-syncd-lock (10) from swss service
Mar 04 03:58:52 sonic root[1891651]: Warm boot flag: swss false.
Mar 04 03:58:52 sonic root[1891651]: Flushing APP, ASIC, COUNTER, CONFIG, and partial STATE databases ...
Mar 04 03:58:53 sonic root[1891651]: Started swss service...
Mar 04 03:58:53 sonic root[1891651]: Unlocking /tmp/swss-syncd-lock (10) from swss service

How to verify it

imuxsock.ratelimit.numratelimiters in syncd should stop continuous growth (or reduce drastically).

Which release branch to backport (provide reason below if selected)

mssonicbld · 2026-03-04T04:05:38Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-04T04:05:49Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull request overview

This PR addresses rsyslogd memory growth in the syncd and swss containers by reducing PID churn that was causing rsyslog's imuxsock ratelimiter to accumulate entries for short-lived senders. Two strategies are applied: suppressing unnecessary output from phc_ctl in phcsync.sh, and anchoring syslog messages to a stable PID ($$) in syncd_common.sh and swss.sh.

Changes:

phcsync.sh now runs phc_ctl with -q -Q flags and redirects stdout to /dev/null to suppress normal output, with explicit error logging on non-zero exit.
syncd_common.sh and swss.sh debug() functions use logger --id=$$ to emit all messages under the parent shell's PID, preventing a new ratelimiter entry per logger invocation.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`platform/mellanox/docker-syncd-mlnx/phcsync.sh`	Adds `-q -Q` flags to silence normal phc_ctl output; redirects only stdout to `/dev/null`, removing the previous `2>/dev/null` stderr suppression
`files/scripts/syncd_common.sh`	Adds `--id=$$` to `logger` in the `debug()` function to anchor all log messages to the parent shell's PID
`files/scripts/swss.sh`	Same `--id=$$` fix as `syncd_common.sh` for the `debug()` function in the swss service script

platform/mellanox/docker-syncd-mlnx/phcsync.sh

tirupatihemanth · 2026-03-04T19:34:25Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2026-03-04T19:34:29Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2026-03-04T19:34:43Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-05T00:08:53Z

/azpw run

mssonicbld · 2026-03-05T00:08:56Z

/AzurePipelines run

azure-pipelines · 2026-03-05T00:09:10Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-05T19:37:28Z

/azpw run

mssonicbld · 2026-03-05T19:37:31Z

/AzurePipelines run

azure-pipelines · 2026-03-05T19:37:45Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-14T00:21:57Z

/azpw run

mssonicbld · 2026-03-14T00:22:00Z

/AzurePipelines run

azure-pipelines · 2026-03-14T00:22:13Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-16T16:01:22Z

/azpw run

mssonicbld · 2026-03-16T16:02:03Z

/AzurePipelines run

azure-pipelines · 2026-03-16T16:02:20Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-17T00:34:43Z

/azpw run

tirupatihemanth · 2026-03-17T00:35:23Z

/azpw run

mssonicbld · 2026-03-17T00:35:56Z

/AzurePipelines run

azure-pipelines · 2026-03-17T00:36:11Z

Azure Pipelines successfully started running 1 pipeline(s).

tirupatihemanth · 2026-03-17T17:31:05Z

/azpw. run

tirupatihemanth · 2026-03-17T17:48:32Z

/azpw. run

Signed-off-by: Hemanth Kumar Tirupati <[email protected]>

mssonicbld · 2026-03-17T19:05:32Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2026-03-17T19:05:46Z

Azure Pipelines successfully started running 1 pipeline(s).

vmittal-msft · 2026-03-19T00:27:45Z

@yxieca @rlhui please help merge

yxieca

Looks good. The PID-stable logger and quiet phc_ctl usage match the known pattern to reduce imuxsock ratelimiter growth. AI agent on behalf of Ying.

mssonicbld · 2026-03-20T20:58:14Z

Cherry-pick PR to 202511: #26298

tirupatihemanth requested a review from lguohan as a code owner March 4, 2026 04:05

Copilot AI review requested due to automatic review settings March 4, 2026 04:05

Copilot started reviewing on behalf of tirupatihemanth March 4, 2026 04:06 View session

vivekrnv approved these changes Mar 4, 2026

View reviewed changes

Copilot AI reviewed Mar 4, 2026

View reviewed changes

platform/mellanox/docker-syncd-mlnx/phcsync.sh Show resolved Hide resolved

Fix rsyslogd memory growth in syncd swss containers

2ef93c5

Signed-off-by: Hemanth Kumar Tirupati <[email protected]>

tirupatihemanth force-pushed the rsyslogd_fix branch from a72dfb4 to 2ef93c5 Compare March 17, 2026 19:05

dgsudharsan added the Request for 202511 Branch label Mar 17, 2026

dgsudharsan approved these changes Mar 18, 2026

View reviewed changes

vmittal-msft added the Approved for 202511 Branch label Mar 19, 2026

yxieca approved these changes Mar 20, 2026

View reviewed changes

yxieca merged commit 468cd8b into sonic-net:master Mar 20, 2026
21 checks passed

mssonicbld mentioned this pull request Mar 20, 2026

[action] [PR:25874] Fix rsyslogd memory growth in syncd swss containers over long term #26298

Merged

6 tasks

mssonicbld added the Created PR to 202511 Branch label Mar 20, 2026

mssonicbld added Included in 202511 Branch and removed Created PR to 202511 Branch labels Mar 21, 2026

Conversation

tirupatihemanth commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Uh oh!

mssonicbld commented Mar 4, 2026

Uh oh!

azure-pipelines bot commented Mar 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

tirupatihemanth commented Mar 4, 2026

Uh oh!

mssonicbld commented Mar 4, 2026

Uh oh!

azure-pipelines bot commented Mar 4, 2026

Uh oh!

tirupatihemanth commented Mar 5, 2026

Uh oh!

mssonicbld commented Mar 5, 2026

Uh oh!

azure-pipelines bot commented Mar 5, 2026

Uh oh!

tirupatihemanth commented Mar 5, 2026

Uh oh!

mssonicbld commented Mar 5, 2026

Uh oh!

azure-pipelines bot commented Mar 5, 2026

Uh oh!

tirupatihemanth commented Mar 14, 2026

Uh oh!

mssonicbld commented Mar 14, 2026

Uh oh!

azure-pipelines bot commented Mar 14, 2026

Uh oh!

tirupatihemanth commented Mar 16, 2026

Uh oh!

mssonicbld commented Mar 16, 2026

Uh oh!

azure-pipelines bot commented Mar 16, 2026

Uh oh!

tirupatihemanth commented Mar 17, 2026

Uh oh!

tirupatihemanth commented Mar 17, 2026

Uh oh!

mssonicbld commented Mar 17, 2026

Uh oh!

azure-pipelines bot commented Mar 17, 2026

Uh oh!

tirupatihemanth commented Mar 17, 2026

Uh oh!

tirupatihemanth commented Mar 17, 2026

Uh oh!

mssonicbld commented Mar 17, 2026

Uh oh!

azure-pipelines bot commented Mar 17, 2026

Uh oh!

vmittal-msft commented Mar 19, 2026

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mssonicbld commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

tirupatihemanth commented Mar 4, 2026 •

edited

Loading