SONiC Container Hardening by Yarden-Z · Pull Request #1364 · sonic-net/SONiC

Yarden-Z · 2023-06-19T10:06:15Z

Create the container hardening initial proposal for SONiC.
During this PR - we would like to present the container hardening criteria and items for SONiC and future requirements in the future for all SONiC containers

Preview:
Containers is a method of creating virtualization and abstraction of an OS for a subset of processes/service on top of a single host with the purpose of giving it an environment to run and execute its tasks without effect of nearby containers/processes.
In SONiC, we are deploying container with full visibility and capabilities as the host Linux.
This poses a security risk and vulnerability as 1 breached container means that the whole system is breached.
Addressing this issue – we have composed this doc for container hardening, containing the security hardening requirements and definitions for all containers on top of SONiC

Repo	PR title	State
sonic-buildimage	[docker-fpm-frr] limit privileged flag for bgp container
sonic-buildimage	[docker-lldp] limit privileged flag for lldp container
sonic-buildimage	[docker-teamd] limit privileged flag for teamd container
sonic-buildimage	[telemetry] limit privileged flag for telemetry container
sonic-buildimage	[docker-mux] limit privileged flag for mux container
sonic-buildimage	[docker-eventd] limit privileged flag for eventd container
sonic-buildimage	[docker-radv] limit privileged flag for radv container
sonic-buildimage	[docker-dhcp-relay] limit privileged flag for dhcp_relay container
sonic-buildimage	[docker-sflow] limit privileged flag for sflow container
sonic-buildimage	[docker-snmp] limit privileged flag for snmp container
sonic-buildimage	limit privileged flag for mgmt-framework container
sonic-buildimage	[docker-restapi] limit privileged flag for restapi container
sonic-buildimage	[docker-macsec] fix privileged and volumes settings
sonic-buildimage	[docker-nat] limit privileged flag for nat container
sonic-buildimage	[docker-p4rt] limit privileged flag for p4rt container
sonic-buildimage	[docker-iccpd] limit privileged flag for iccpd container
sonic-buildimage	Container net host remove
sonic-snmpagent	SNMP hardening - redis socket connect
sonic-buildimage	[docker-database] limit privileged flag for database container

sonic-mgmt PRs

Repo	PR title	State
sonic-mgmt	testcase for bgp container in container hardening
sonic-mgmt	[container hardening] include multiasic scenario, add lldp and teamd containers to testcase
sonic-mgmt	[container hardening] include more containers to test case

Revert related PR

Repo	PR title	State
sonic-buildimage	Remove privileged flag for database and snmp docker
sonic-buildimage	Revert "Remove privileged flag for database and snmp docker"

syncd, swss cannot be reduced privileged

linux-foundation-easycla · 2023-06-19T10:06:22Z

The committers listed above are authorized under a signed CLA.

✅ login: Yarden-Z (80c9557, 67afbf0, e971302, 2a3964d, e4e5c05, 4ce3232)
✅ login: maipbui / name: Mai Bui (9469782, 29e3da4, edac201)

Signed-off-by: Mai Bui <[email protected]>

doc/Container Hardening/SONiC_container_hardening_HLD.md

maipbui · 2023-07-03T17:34:19Z

@Yarden-Z I made some changes directly to the HLD, please check again

venkatmahalingam · 2023-07-03T17:50:42Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+
+### 3.2 net=host
+
+Removing the `net=host` is required to prevent the container from accessing the full network scope of the host and system.


I believe, BGP container needs full network access in order to function properly, do we have the list of containers that can be migrated without net=host? Some of the container (DHCP relay, SNMP..etc) directly work on the Linux networking, removing net=host, might require additional changes at the application level.

On the SNMP container we have managed to remove the net=host.
For BGP - we also have a PR up with @maipbui changes that we can review.

For DHCP - we'll need to check this.
I have a composed list of containers and the suggestions on whether we can remove the flags or not, I'll add it to the HLD as an appendix.

Please help review code PRs which we included in this HLD PR's description. @venkatmahalingam Yes, BGP container needs host network mode.

Signed-off-by: Mai Bui <[email protected]>

Blueve · 2023-07-04T03:51:13Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+
+### 4.2 net=host optimization
+
+Here we will provide a detailed example of how to switch from the `--net=host` configuration (host network) to the `--net=bridge` configuration paired with port forwarding in a specific container. We are using the database container as an example for this item.


Will this design cover SDK change? SONiC containers are massively interactive with Redis database, it is required to let our SDK adapt bridge mode container

Why does this require SDK changes?
Regarding the SDK - since it resides in a single container, we have decided against reducing the privileges and capabilities of this container, as this will require massive SDK re-design (and this is not our purpose).
Also - the SDK container is not at a high risk of compromise since it does not interact with the outside network.

@Blueve Which SDK? Is it switch ASIC vendor SDK or swss-common library?
Could you give specific sample function in the SDK you are worrying about?

@qiluo-msft swss-common, do we still have a unified way to interact with Redis database with this design?

Current we can use two approaches to interact Redis in sdk:

local IP

Unix sock file

With this design, both of them are not available I think.

Both are still applicable with this design (tested).
UDS should not be affected, since reducing the privileges of the redis container does not affect the Unix socket (on a different level of permissions - file access permissions).
Local IP might require a slight change, but you can check it in the relevant PR and it is currently working.

Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): 14807420 How I did it Reduce linux capabilities in privileged flag, retain NET_ADMIN and SYS_ADMIN capabilities How to verify it Install new image to DUT, verify bgp container is up Run bgp sonic-mgmt kvmtest

Description of PR HLD implementation: Container Hardening (sonic-net/SONiC#1364) Dependency: sonic-net/sonic-buildimage#14932 #### What is the motivation for this PR? Check bgp container has access to /dev/sda* or /dev/vda* after limiting privileged flag to less Linux capabilities. #### How did you do it? #### How did you verify/test it? ``` container_hardening/test_container_hardening.py::test_bgp_dev PASSED [100%] ``` Signed-off-by: Mai Bui <[email protected]>

#### Why I did it Reduce docker privilege This is part of HLD sonic-net/SONiC#1364 #### How I did it Remove flag --privileged #### How to verify it docker exec -it database bash root@0048b82b460b:/# ip link add dummy0 type dummy RTNETLINK answers: Operation not permitted

#### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag, retain NET_ADMIN capability

lguohan · 2023-08-17T19:09:02Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+| SYS_PTRACE                    | Trace arbitrary processes using ptrace(2).        |
+| SYS_RAWIO                             | Perform I/O port operations (iopl(2) and ioperm(2)).         |
+| SYS_RESOURCE                  | Override resource Limits        |
+| SYS_TIME                              | Set system clock (settimeofday(2), stime(2), adjtimex(2)); set real-time (hardware) clock.         |


how do we know if the appliation does not use SYS_TIME?

We don't, but if a specific container requires this - it can add it as a specific flag only for this container.

venkatmahalingam · 2023-09-11T17:29:54Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+## 10. Open/Action items - if any
+
+Currently, Nvidia and MSFT have scoped commitment for specific containers.
+Redis and SNMP already have these adjustments.


What if SNMP container wants to listen on some particular interface on the host? container hardening will be disabled by default for now? can this be driven based on FEATURE table config?

If the SNMP container would like to listen to a specific interface - you can check this:
https://github.com/sonic-net/sonic-buildimage/pull/15176/files

I think that this PR will showcase your specific use-case.

zhangyanzhao · 2023-09-12T15:10:31Z

Community review recording https://zoom.us/rec/share/jwTKj7zNviREe488pYQugvH4dIaUFXVYs1N1iDv5jzjnFbVq1TT09vqbrwDIIski.Z6KNg_je7AawlH2V

doc/Container Hardening/SONiC_container_hardening_HLD.md

madhupalu · 2023-09-14T22:45:59Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+					-p 6379:6379 \
+	{%- endif %}
+
+#### How to check?


What is the user experience with this change with the current container capabilities? What is the plan to mitigate this?

User should not experience any changes.
If there are, then one of the following should occur:

Adjust the user action/container action so that it does not require the elevated privileges or specific net host capabilities.

Add to the container the required capabilities it needs to perform.

We should go with option 1 first, and only after we have decided option 1 is not feasible - option 2 is on the table.

madhupalu · 2023-09-14T22:46:15Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+N/A, this feature will be checked on a system level.
+
+### 9.2 System Test cases
+


Is there any impact w.r.t memory and cpu as it is running in different network name spaces?

Negligible, as it is part of the infra.
I'll be more precise - the impact is as much of how much the Linux ACL takes, but this should not be visible to any user.

madhupalu · 2023-09-14T22:47:03Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+
+### 3.1 Root privileges
+
+When removing the root privileges from a specific container - we are required to remove the `--privileged` flag and add the required missing Linux capabilities to the docker, or alternatively adjust the container so that it does not require root privileges to perform any action.


There is similar work done with Multi ASIC design, it is better to converge before making any decision changing net=host -> bridge etc.

Not familiar with it, what is the work being done there?

doc/Container Hardening/SONiC_container_hardening_HLD.md

madhupalu · 2023-09-14T22:47:47Z

doc/Container Hardening/SONiC_container_hardening_HLD.md

+N/A
+
+### 6.2. CLI/YANG model Enhancements
+


Is there a tool that knows what containers are running privileged mode?

There might be, requires a bit of research.

maybe you can check by docker inspect container_name | grep Privileged?

This might be applicable for the --privileged flag, but this HLD is also discussing host network.
So while this might work for one of the HLD item (removing containers from privilege mode) it does not scope the whole feature list.
So this command might be a good start, but we should look at a more encompassing approach to check this (and not stop here).

…iner [docker-acms] limit privileged flag for acms container #### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Run acms sonic-mgmt tests Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-01:~$ docker inspect acms | grep Privilege "Privileged": false, admin@vlab-01:~$ docker exec -it acms bash root@vlab-01:/# capsh --print Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep ``` Signed-off-by: Mai Bui <[email protected]>

liat-grozovik · 2024-02-12T13:59:38Z

@zhangyanzhao this HLD has been shared long time ago and most of the code PRs were already merged.
I believe there was enough time to provide feedback. as comments were addressed and IMO it is time to merge this PR and ensure the rest of the PRs get in master soon

Added note regarding user-defined bridges. Not in the scope of this HLD at the moment

#### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: 14807420 #### How I did it Reduce linux capabilities in privileged flag #### How to verify it Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. ``` admin@vlab-08:~$ docker inspect database0 | jq '.[0] | {CapAdd: .HostConfig.CapAdd, Privileged: .HostConfig.Privileged, AppArmor: .AppArmorProfile, Security Opt: .HostConfig.SecurityOpt, Devices: .HostConfig.Devices, MaskedPath: .HostConfig.MaskedPaths, ReadonlyPaths: .HostConfig.ReadonlyPaths}' { "CapAdd": null, "Privileged": false, "AppArmor": "unconfined", "SecurityOpt": [ "apparmor=unconfined" ], "Devices": [], "MaskedPath": [], "ReadonlyPaths": [] } ```

Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) How I did it Reduce linux capabilities in privileged flag How to verify it Run platform tests. Check container's settings: Privileged is false and container only has default Linux caps, and SYS_RAWIO/SYS_ADMIN cap.

qiluo-msft · 2025-11-17T21:05:22Z

New gnmi container:
Track the issue in sonic-net/sonic-buildimage#24542

Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) How I did it Reduce linux capabilities in privileged flag How to verify it Run platform tests. Check container's settings: Privileged is false and container only has default Linux caps, and SYS_RAWIO/SYS_ADMIN cap. Signed-off-by: Feng Pan <[email protected]>

Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) Work item tracking Microsoft ADO (number only): How I did it How to verify it Run otel sonic-mgmt tests admin@vlab-01:~$ docker inspect otel | grep Privi "Privileged": false, Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps. Signed-off-by: Janet Cui <[email protected]>

#### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it #### How to verify it Run otel sonic-mgmt tests ``` admin@vlab-01:~$ docker inspect otel | grep Privi "Privileged": false, ``` Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps.  #### Which release branch to backport (provide reason below if selected)  - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [x] 202511 #### Tested branch (Please provide the tested image version) 202412  - [ ]  - [ ]  #### Description for the changelog   #### Link to config_db schema for YANG module changes  Signed-off-by: Sonic Build Admin <[email protected]> #### A picture of a cute animal (not mandatory but encouraged)

…ainer (#2053)  #### Why I did it HLD implementation: Container Hardening (sonic-net/SONiC#1364) ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it #### How to verify it Run otel sonic-mgmt tests ``` admin@vlab-01:~$ docker inspect otel | grep Privi "Privileged": false, ``` Check container's settings: Privileged is false and container only has default Linux caps, does not have extended caps.  #### Which release branch to backport (provide reason below if selected)  - [ ] 202305 - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [x] 202511 #### Tested branch (Please provide the tested image version) 202412  - [ ]  - [ ]  #### Description for the changelog   #### Link to config_db schema for YANG module changes  Signed-off-by: Sonic Build Admin <[email protected]> #### A picture of a cute animal (not mandatory but encouraged)

Yarden-Z added 4 commits June 19, 2023 12:56

Add files via upload

e971302

Merge branch 'sonic-net:master' into master

67afbf0

Create SONiC Container Hardening

80c9557

Delete SONiC Container Hardening.md

2a3964d

This was referenced Jun 26, 2023

[docker-fpm-frr] limit privileged flag for bgp container sonic-net/sonic-buildimage#14932

Merged

testcase for bgp container in container hardening sonic-net/sonic-mgmt#8694

Merged

maipbui added 2 commits July 3, 2023 17:22

Modify HLD

29e3da4

Signed-off-by: Mai Bui <[email protected]>

remove old file

edac201

Signed-off-by: Mai Bui <[email protected]>

maipbui reviewed Jul 3, 2023

View reviewed changes

doc/Container Hardening/SONiC_container_hardening_HLD.md Show resolved Hide resolved

venkatmahalingam reviewed Jul 3, 2023

View reviewed changes

fix

9469782

Signed-off-by: Mai Bui <[email protected]>

Blueve reviewed Jul 4, 2023

View reviewed changes

This was referenced Jul 13, 2023

[docker-teamd] limit privileged flag for teamd container sonic-net/sonic-buildimage#15829

Merged

[docker-lldp] limit privileged flag for lldp container sonic-net/sonic-buildimage#15830

Merged

qiluo-msft mentioned this pull request Aug 3, 2023

Remove privileged flag for database and snmp docker sonic-net/sonic-buildimage#13783

Merged

8 tasks

lguohan reviewed Aug 17, 2023

View reviewed changes

maipbui mentioned this pull request Sep 7, 2023

[telemetry] limit privileged flag for telemetry container sonic-net/sonic-buildimage#16350

Merged

11 tasks

venkatmahalingam reviewed Sep 11, 2023

View reviewed changes

maipbui mentioned this pull request Sep 14, 2023

[docker-mux] limit privileged flag for mux container sonic-net/sonic-buildimage#16562

Merged

11 tasks

madhupalu reviewed Sep 14, 2023

View reviewed changes

Update SONiC_container_hardening_HLD.md

4ce3232

Added note regarding user-defined bridges. Not in the scope of this HLD at the moment

qiluo-msft approved these changes Feb 26, 2024

View reviewed changes

maipbui approved these changes Feb 26, 2024

View reviewed changes

liat-grozovik merged commit 417a096 into sonic-net:master Feb 29, 2024

maipbui mentioned this pull request Jul 17, 2024

[docker-database] limit privileged flag for database container sonic-net/sonic-buildimage#19248

Merged

11 tasks

nser77 mentioned this pull request Oct 26, 2024

add vrrpmgrd to support FRR-VRRP configuration sonic-net/sonic-swss#3106

Open

DavidZagury mentioned this pull request Jul 13, 2025

[docker-pmon] limit privileged flag for pmon container DavidZagury/sonic-buildimage#78

Closed

9 tasks

DavidZagury mentioned this pull request Jul 24, 2025

[docker-pmon] limit privileged flag for pmon container sonic-net/sonic-buildimage#23457

Merged

9 tasks

This was referenced Nov 17, 2025

Add privileged mode and host access to gnmi container for gNOI operations sonic-net/sonic-buildimage#24056

Merged

gnmi container should remove privileged mode sonic-net/sonic-buildimage#24542

Closed

Janetxxx mentioned this pull request Mar 6, 2026

[docker-otel] limit privileged flag for otel container sonic-net/sonic-buildimage#25930

Merged

8 tasks

mssonicbld mentioned this pull request Mar 12, 2026

[action] [PR:25930] [docker-otel] limit privileged flag for otel container Azure/sonic-buildimage-msft#2053

Merged

8 tasks

mssonicbld mentioned this pull request Mar 13, 2026

[action] [PR:25930] [docker-otel] limit privileged flag for otel container sonic-net/sonic-buildimage#26159

Merged

8 tasks


		### 3.2 net=host

		Removing the `net=host` is required to prevent the container from accessing the full network scope of the host and system.


		### 4.2 net=host optimization

		Here we will provide a detailed example of how to switch from the `--net=host` configuration (host network) to the `--net=bridge` configuration paired with port forwarding in a specific container. We are using the database container as an example for this item.

		N/A, this feature will be checked on a system level.

		### 9.2 System Test cases


		### 3.1 Root privileges

		When removing the root privileges from a specific container - we are required to remove the `--privileged` flag and add the required missing Linux capabilities to the docker, or alternatively adjust the container so that it does not require root privileges to perform any action.

Conversation

Yarden-Z commented Jun 19, 2023 • edited by maipbui Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

maipbui commented Jul 3, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Blueve Jul 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft Jul 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

venkatmahalingam Sep 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhangyanzhao commented Sep 12, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liat-grozovik commented Feb 12, 2024

Uh oh!

qiluo-msft commented Nov 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

Yarden-Z commented Jun 19, 2023 •

edited by maipbui

Loading

linux-foundation-easycla bot commented Jun 19, 2023 •

edited

Loading

Blueve Jul 4, 2023 •

edited

Loading

qiluo-msft Jul 5, 2023 •

edited

Loading

venkatmahalingam Sep 11, 2023 •

edited

Loading