[docker-pmon] limit privileged flag for pmon container by DavidZagury · Pull Request #23457 · sonic-net/sonic-buildimage

DavidZagury · 2025-07-24T07:18:51Z

Why I did it

HLD implementation: Container Hardening (sonic-net/SONiC#1364)

Work item tracking

Microsoft ADO (number only):

How I did it

Reduce linux capabilities in privileged flag

How to verify it

Run platform tests.
Check container's settings: Privileged is false and container only has default Linux caps, and SYS_RAWIO/SYS_ADMIN cap.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

mssonicbld · 2025-07-24T07:18:58Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2025-07-24T07:19:11Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-30T08:04:29Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2025-07-30T08:04:43Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-31T07:43:36Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2025-07-31T07:43:48Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-04T10:53:42Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2025-08-04T10:53:55Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-08-04T15:09:02Z

/azp run Azure.sonic-buildimage

azure-pipelines · 2025-08-04T15:09:16Z

Azure Pipelines successfully started running 1 pipeline(s).

adyeung · 2025-08-11T18:08:23Z

@ds952811 pls help review

qiluo-msft · 2025-08-11T18:14:10Z

@adyeung Could you help review and verify the container hardening of pmon work for Broadcom asic platform?

qiluo-msft

LGTM, but need other platform vendors to test

Why I did it For platform using sysfs, the current pmon is broken. admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: [Errno 30] Read-only file system: '/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' After this fix admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) <_io.FileIO name='/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' mode='rb+' closefd=True> This regression was introduced by #23457

nonodark · 2025-09-18T00:31:53Z

Hi @DavidZagury and @qiluo-msft,
Could you also import the IPMI interface (/dev/ipmi0) into the pmon Docker container?
Many services, such as thermald and psud, may rely on this interface to get sensor information.

qiluo-msft · 2025-09-18T00:34:36Z

Hi @DavidZagury and @qiluo-msft, Could you also import the IPMI interface (/dev/ipmi0) into the pmon Docker container? Many services, such as thermald and psud, may rely on this interface to get sensor information.

Feel free to raise PRs if it helps. I guess @DavidZagury have no such platforms to test.

Why I did it On platforms using the IPMI interface, the current pmon is broken. root@sonic:~# show logging "Could not open device" 2025 Sep 18 02:18:30.578482 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:18:31.689267 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:55.434346 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:57.355865 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:57.355865 sonic INFO pmon#supervisord: thermalctld Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:58.653110 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory root@sonic:~# docker exec -it pmon bash -c "ls /dev/ipmi*" ls: cannot access '/dev/ipmi*': No such file or directory root@sonic:~# show platform psustatus Error: Failed to get PSU status Error: failed to get PSU status from state DB root@sonic:~# show platform temperature Thermal Not detected root@sonic:~# After this fix root@sonic:~# show logging "Could not open device" root@sonic:~# docker exec -it pmon bash -c "ls /dev/ipmi*" /dev/ipmi0 root@sonic:~# show platform psustatus PSU Model Serial HW Rev Voltage (V) Current (A) Power (W) Status LED ----- ---------- ------------------- -------- ------------- ------------- ----------- -------- ----- PSU1 YNEE0750EM F7510AS90580X020027 N/A 0.00 0.00 0.00 NOT OK green PSU2 YNEE0750EM F7510AS90580X020024 N/A 12.12 10.00 121.20 OK green root@sonic:~# show platform temperature Sensor Temperature High TH Low TH Crit High TH Crit Low TH Warning Timestamp ---------------- ------------- --------- -------- -------------- ------------- --------- ----------------- PSU1_TEMP1 32 N/A N/A N/A N/A False 20250918 02:07:24 PSU2_TEMP1 32 N/A N/A N/A N/A False 20250918 02:07:24 TEMP_ENV_BMC 37 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_ENV_MACCASE 40 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_ENV_PSUCASE 32 57.0 N/A 62.0 N/A False 20250918 02:07:23 TEMP_ENV_SSDCASE 42 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_MAC 41 95.0 N/A 105.0 N/A False 20250918 02:07:22 TEMP_PSU0_TEMP1 32 N/A N/A 70.0 N/A False 20250918 02:07:23 TEMP_PSU1_TEMP1 32 N/A N/A 70.0 N/A False 20250918 02:07:24 root@sonic:~# This regression was introduced by #23457

Pavan-Nokia · 2025-09-25T18:41:54Z

@DavidZagury @qiluo-msft @prgeor

With this PR included we see that the privileges for SYSFS have been made read-only in the PMON docker

root@sonic:/# mount |grep sys
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)

The thermactld running inside the PMON docker is unable to control the fan speed as it cannot write into the PWM register and the devices are going into overheat

admin@sonic:~$ docker exec -it pmon bash
root@sonic:/# cat /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1
128
root@sonic:/# echo 140 > /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1
bash: /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1: Read-only file system

qiluo-msft · 2025-09-25T19:13:25Z

@DavidZagury @qiluo-msft @prgeor

With this PR included we see that the privileges for SYSFS have been made read-only in the PMON docker
root@sonic:/# mount |grep sys
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
The thermactld running inside the PMON docker is unable to control the fan speed as it cannot write into the PWM register and the devices are going into overheat
admin@sonic:~$ docker exec -it pmon bash
root@sonic:/# cat /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1
128
root@sonic:/# echo 140 > /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1
bash: /sys/bus/i2c/devices/17-002d/hwmon/hwmon5/pwm1: Read-only file system

Could you share the mount part in docker inspect pmon ?

Pavan-Nokia · 2025-09-25T19:35:06Z

docker inspect pmon

I had not included #24017 in my build. That PR may address the issue.
I will try out a new build with that included

…-net#23457)" This reverts commit a13685f.

Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) How I did it Reduce linux capabilities in privileged flag How to verify it Run platform tests. Check container's settings: Privileged is false and container only has default Linux caps, and SYS_RAWIO/SYS_ADMIN cap. Signed-off-by: Feng Pan <fenpan@microsoft.com>

Why I did it For platform using sysfs, the current pmon is broken. admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: [Errno 30] Read-only file system: '/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' After this fix admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) <_io.FileIO name='/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' mode='rb+' closefd=True> This regression was introduced by sonic-net#23457 Signed-off-by: Feng Pan <fenpan@microsoft.com>

Why I did it On platforms using the IPMI interface, the current pmon is broken. root@sonic:~# show logging "Could not open device" 2025 Sep 18 02:18:30.578482 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:18:31.689267 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:55.434346 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:57.355865 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:57.355865 sonic INFO pmon#supervisord: thermalctld Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory 2025 Sep 18 02:21:58.653110 sonic INFO pmon#supervisord: psud Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory root@sonic:~# docker exec -it pmon bash -c "ls /dev/ipmi*" ls: cannot access '/dev/ipmi*': No such file or directory root@sonic:~# show platform psustatus Error: Failed to get PSU status Error: failed to get PSU status from state DB root@sonic:~# show platform temperature Thermal Not detected root@sonic:~# After this fix root@sonic:~# show logging "Could not open device" root@sonic:~# docker exec -it pmon bash -c "ls /dev/ipmi*" /dev/ipmi0 root@sonic:~# show platform psustatus PSU Model Serial HW Rev Voltage (V) Current (A) Power (W) Status LED ----- ---------- ------------------- -------- ------------- ------------- ----------- -------- ----- PSU1 YNEE0750EM F7510AS90580X020027 N/A 0.00 0.00 0.00 NOT OK green PSU2 YNEE0750EM F7510AS90580X020024 N/A 12.12 10.00 121.20 OK green root@sonic:~# show platform temperature Sensor Temperature High TH Low TH Crit High TH Crit Low TH Warning Timestamp ---------------- ------------- --------- -------- -------------- ------------- --------- ----------------- PSU1_TEMP1 32 N/A N/A N/A N/A False 20250918 02:07:24 PSU2_TEMP1 32 N/A N/A N/A N/A False 20250918 02:07:24 TEMP_ENV_BMC 37 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_ENV_MACCASE 40 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_ENV_PSUCASE 32 57.0 N/A 62.0 N/A False 20250918 02:07:23 TEMP_ENV_SSDCASE 42 75.0 N/A 80.0 N/A False 20250918 02:07:23 TEMP_MAC 41 95.0 N/A 105.0 N/A False 20250918 02:07:22 TEMP_PSU0_TEMP1 32 N/A N/A 70.0 N/A False 20250918 02:07:23 TEMP_PSU1_TEMP1 32 N/A N/A 70.0 N/A False 20250918 02:07:24 root@sonic:~# This regression was introduced by sonic-net#23457 Signed-off-by: Feng Pan <fenpan@microsoft.com>

qiluo-msft · 2026-01-16T01:59:30Z

files/build_templates/docker_image_ctl.j2

        -e SX_API_SOCKET_FILE=/var/run/sx_sdk/sx_api.sock \
 {%- elif docker_container_name == "pmon" %}
+        -v /sys/devices/platform/mlxplat:/sys/devices/platform/mlxplat:rw \
+        -v /sys/module/sx_core:/sys/module/sx_core:rw \


This path may not exist on Mellanox platform.

admin@sonic:~$ docker restart pmon Error response from daemon: Cannot restart container pmon: error while creating mount source path '/sys/module/sx_core': mkdir /sys/module/sx_core: operation not permitted

gregoryboudreau · 2026-01-21T19:01:45Z

@qiluo-msft @DavidZagury this change has resulted in issues within PMON having access to monitor/interact with devices under /dev/ (#25142). We are currently investigating some ways to handle this, in the SONiC Chassis weekly call this was discussed and we wanted to get some further insight into the background and requirements of this change as well as potential solutions impacts. @rlhui to help setup a call to discuss this further, thanks!

Why I did it For platform using sysfs, the current pmon is broken. admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: [Errno 30] Read-only file system: '/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' After this fix admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) <_io.FileIO name='/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' mode='rb+' closefd=True> This regression was introduced by sonic-net#23457

Why I did it For platform using sysfs, the current pmon is broken. admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) Traceback (most recent call last): File "<stdin>", line 1, in <module> OSError: [Errno 30] Read-only file system: '/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' After this fix admin@sonic:~$ docker exec -it pmon bash root@sonic:/# python3 Python 3.11.2 (main, Apr 28 2025, 14:11:48) [GCC 12.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> open('/sys/class/i2c-adapter/i2c-71/71-0050/eeprom', mode='r+b', buffering=0) <_io.FileIO name='/sys/class/i2c-adapter/i2c-71/71-0050/eeprom' mode='rb+' closefd=True> This regression was introduced by sonic-net#23457 Signed-off-by: Boyang Yu <byu@arista.com>

DavidZagury added 8 commits July 2, 2025 18:19

Test with most permissions remove

e4a26a6

Remove apparmor and systempaths restrictions

12239a2

Fix missing dev/sda and sys/module

4afc1d5

Add watchdogs

36add0d

Fix docker_image_ctl

b2983ed

No need for SYS_MODULE

ce85c25

Move mlx paths to only be on mlx

8d4c248

Remove comment

4a69672

Remove sda

acd72c6

DavidZagury force-pushed the master_pmon_harden branch from bb28663 to acd72c6 Compare July 31, 2025 07:43

DavidZagury force-pushed the master_pmon_harden branch from cb29428 to acd72c6 Compare August 4, 2025 15:08

qiluo-msft requested review from adyeung, maipbui, prgeor and vvolam August 11, 2025 18:04

qiluo-msft requested a review from judyjoseph August 11, 2025 18:12

qiluo-msft reviewed Aug 11, 2025

View reviewed changes

maipbui approved these changes Aug 15, 2025

View reviewed changes

DavidZagury requested a review from lguohan as a code owner September 2, 2025 08:21

qiluo-msft approved these changes Sep 9, 2025

View reviewed changes

qiluo-msft merged commit a13685f into sonic-net:master Sep 9, 2025
23 checks passed

qiluo-msft mentioned this pull request Sep 17, 2025

Add sysfs mount for pmon container #24017

Merged

9 tasks

nonodark mentioned this pull request Sep 18, 2025

Add ipmi device for pmon container #24037

Merged

9 tasks

Pavan-Nokia mentioned this pull request Oct 12, 2025

[docker-pmon] Add ssd and i2c0 for pmon container #24250

Merged

9 tasks

rohinikumart mentioned this pull request Oct 15, 2025

Bug:[docker-pmon]xcvrd fails to come up in dell platforms #24271

Closed

stepanblyschak pushed a commit to stepanblyschak/sonic-buildimage that referenced this pull request Nov 14, 2025

Revert "[docker-pmon] limit privileged flag for pmon container (sonic…

cfdc756

…-net#23457)" This reverts commit a13685f.

This was referenced Dec 19, 2025

Add device mount for pmon container #24907

Closed

Add device mount for pmon container #24943

Closed

qiluo-msft reviewed Jan 16, 2026

View reviewed changes

rlhui mentioned this pull request Jan 21, 2026

Bug: PMON missing access to necessary devices for platform monitoring post privilege removal #25142

Open

micas-net mentioned this pull request Mar 5, 2026

Bug:add missing devices due to unprivileged mode #25913

Open

byu343 mentioned this pull request Mar 24, 2026

[202505] Add sysfs mount for pmon container (#24017) #26346

Open

8 tasks

Conversation

DavidZagury commented Jul 24, 2025

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

mssonicbld commented Jul 24, 2025

Uh oh!

azure-pipelines bot commented Jul 24, 2025

Uh oh!

mssonicbld commented Jul 30, 2025

Uh oh!

azure-pipelines bot commented Jul 30, 2025

Uh oh!

mssonicbld commented Jul 31, 2025

Uh oh!

azure-pipelines bot commented Jul 31, 2025

Uh oh!

mssonicbld commented Aug 4, 2025

Uh oh!

azure-pipelines bot commented Aug 4, 2025

Uh oh!

mssonicbld commented Aug 4, 2025

Uh oh!

azure-pipelines bot commented Aug 4, 2025

Uh oh!

adyeung commented Aug 11, 2025

Uh oh!

qiluo-msft commented Aug 11, 2025

Uh oh!

qiluo-msft left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nonodark commented Sep 18, 2025

Uh oh!

qiluo-msft commented Sep 18, 2025

Uh oh!

Pavan-Nokia commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

qiluo-msft commented Sep 25, 2025

Uh oh!

Pavan-Nokia commented Sep 25, 2025

Uh oh!

qiluo-msft Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

gregoryboudreau commented Jan 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Pavan-Nokia commented Sep 25, 2025 •

edited

Loading