Multi-ASIC implementation by SuvarnaMeenakshi · Pull Request #3888 · sonic-net/sonic-buildimage

SuvarnaMeenakshi · 2019-12-12T20:11:06Z

- What I did
To support SONiC on multi-asic platforms, the approach is to create SONiC docker stack for each ASIC. Each SONiC docker stack is separated by network namespaces for each ASIC.
Each network namespace will comprise of key dockers :
database
swss
syncd
bgp
teamd
interfaces-config
lldp

- How I did it
Main changes done to support SONiC on multi-asic platform are:

systemd service template for multi-instance service is added to start N number of systemd services where N is the number of ASICs in the platform.
Changes in FRR config to support BGP between the ASIC namespaces. To support this, a new type is added in DEVICE_METADATA called "InternalFrontend" and "InternalBackend" to differentiate between a front-end ASIC and back-end ASIC.
Change in docker_img_ctl.j2 to support starting dockers with a specific ASIC instance number which will be passed to the <docker_name>.sh script by the systemd service.
Apart from multiple instances of database docker, a global database docker will exist in the host namespace.

- How to verify it
Build a multi-asic VS image.
Follow instruction (in README.vsd) to bring up multi-asic VS image by adding the number of ASICs in device/platform/asic.conf and also adding a specific topology.sh in HWSKU directory to create a virtual ASIC connectivity in the VS.
Once the VS image comes up, ensure the dockers come up and verify BGP sessions between ASICs.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

* create systemd target to start/stop all containers in a specific namespace at once Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add multi-instance option to redis-cli * create database template service * change docker_image_ctl to use new redis sockets * increase database service timeout duration to 120s Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add instance number option and unix socket option to docker_image_ctl for swss * change swss service to service template * add instance number option and unix socket option to swss.sh * force all redis-cli calls through redis-cli wrapper Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

* add syncd service template * add instance number argument for syncd script

* add rsyslog service template compatible with network namespaces * change rsyslog_config to use database instance unix socket Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit e871f11)

* add wrapper for 'sudo ip netns exec' command Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 85745b7)

* increase from 2GB to 8GB to prevent OOM crashes * necessary because multi-ASIC devices have increased memory requirements Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 845778e)

* add bgp service template * create /etc/sonic/frr* for multiple instances of bgp docker container [frr docker]: add multi-ASIC support to bgpd.conf * add `redistribute connected` option to advertise routes to local interfaces * add `all` option to next-hop-self * add route-map to hide internal routes * add route-map to allow duplicate frontend router id Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 4ea01b7)

* change teamd service file to template * force teamd to restart when swss does Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 0297279)

* set submodule to point at newest commit Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit ebab66d)

* create service template * enable template instances in sonic_debian_extension * use iproute2 in interfaces-config script if working with network namespaces * make swss multi-instance dependent on multi-instance interfaces config Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 08af0db)

* use 'asic' as namespace prefix instead of 'namespace' in accordance with PR sonic-net#3269 Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit c1444e3)

lguohan · 2020-01-29T19:26:26Z

@pavel-shirshov , can you check the bgp template?

lguohan · 2020-01-29T19:30:14Z

files/image_config/rsyslog/rsyslog-config@.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=Update rsyslog configuration


i do not understand why when need multi-instance for rsyslog.

Removed multiple instances for rsyslog service. Changes required for rsyslog service support for multi-asic platform will be added in a different PR.

lguohan · 2020-01-29T19:31:27Z

files/scripts/swss.sh

    if [[ x"$WARM_BOOT" != x"true" ]]; then
-        /bin/systemctl start ${PEER}
+        if [[ ! -z $DEV ]]; then
+                       /bin/systemctl start ${PEER}@$DEV


need consistent spacing.

abdosi · 2020-01-30T22:22:04Z

files/build_templates/multi_instance/bgp@.service.j2

@@ -0,0 +1,16 @@
+[Unit]
+Description=BGP container
+#Requires=updategraph.service


@SuvarnaMeenakshi @lguohan : we don't need this dependency for Multi Instance ? Single Instance still have this dependency.

support for updategraph and config-setup service for multi-asic platform will be added in a separate PR. As updategraph service is not updated to support multiple config_db, this dependency cannot be added here.

qiluo-msft · 2020-01-31T02:19:33Z

Please solve the conflicts #Closed

dockers/docker-database/base_image_files/redis-cli

different namespaces.

database instances

database instances.

to talk to different database instances

for multi-asic platform

platform

SuvarnaMeenakshi · 2020-03-19T18:30:39Z

retest vsimage please

paulnice

Looks good for me. Please wait for others.

lguohan · 2020-03-20T02:57:00Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-20T17:21:46Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-21T01:18:42Z

retest vsimage please

lguohan · 2020-03-24T07:10:22Z

pending on vsimage test passing.

SuvarnaMeenakshi · 2020-03-26T18:18:35Z

retest vsimage please

SuvarnaMeenakshi · 2020-03-31T06:15:04Z

retest vsimage please

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

This reverts commit 2e87a16.

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

This reverts commit 2e87a16.

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

wendani · 2020-05-09T05:35:52Z

Multi-ASIC is not part of the 201911 release https://github.com/Azure/SONiC/wiki/Release-Progress-Tracking-201911

Why do we include it in 201911 @abdosi ?

…atically (#22686) #### Why I did it src/sonic-utilities ``` * e18640e - (HEAD -> master, origin/master, origin/HEAD) Switchport mode update for 'show interfaces status' (#3788) (3 hours ago) [Shivashankar C R] * 809646a - Revert "Addition of prober_type in config and show commands for muxcable (#3884)" (#3979) (17 hours ago) [Xin Wang] * 3db35d5 - `vnet_route_check.py` should not report VNET routes in APP DB but not in STATE DB and ASIC DB as mismatches (#3990) (26 hours ago) [mramezani95] * 8647356 - [show][config][plugin] add processing of ModuleNotFoundError with log_warning (#3832) (32 hours ago) [Maksym Kovalchuk] * 20976de - fix show bgp cli on multiple asic device (#3981) (5 days ago) [Liping Xu] * 46c82ab - [db_migrator] Fix parse_xml fails when minigraph has SonicQosProfile (#3972) (6 days ago) [Xin Wang] * 1c3f789 - Fix route_check.py to ignore local p2p IP prefixes (#3882) (7 days ago) [prabhataravind] * 898a037 - Make 'show interface errors' lookup the correct oper_error_status count published by OA (#3956) (8 days ago) [Bobby McGonigle] * eda6ada - [sonic-package-manager] Save tag that was used to install the application (#3917) (8 days ago) [DavidZagury] * c409594 - [SPM] Add support for configuring systemd service Type in package manifests (#3946) (8 days ago) [DavidZagury] * 09b4292 - [trim]: Add Packet Trimming Asym DSCP CLI (#3920) (9 days ago) [Nazarii Hnydyn] * f751730 - Lodoga-Prime: lodogaprime platform support (#3954) (13 days ago) [NobutomoNakano] * 0424ae0 - Add GCU Support for SKU Mellanox-SN4280-C48/O8C40/O8V40 (#3964) (13 days ago) [Sai Rama Mohan Reddy S] * 57b9846 - fix issue #22476 remove quagga in show bgp cmd (#3947) (2 weeks ago) [Liping Xu] * 5d11fc5 - Fix comparison error when replace (#3941) (3 weeks ago) [jingwenxie] * f6d6d9a - Fix for 22138: Chassisd does not wait for the execution to complete for previous admin state change requests - Replaces PR: #3845 (#3937) (4 weeks ago) [rameshraghupathy] * be72304 - [YANG] remove uses clause handling, now part of sonic-yang-mgmt (#3814) (4 weeks ago) [Brad House] * 19a6b3c - Switch to using chrony instead of ntpd : gcu - services_validator.py (#3929) (4 weeks ago) [Anukul Verma] * 5db9c27 - Fixed cli command for ECN config on voq switch (#3928) (4 weeks ago) [saksarav-nokia] * 020f9d0 - Improved GCU's field validation logic for the WRED_PROFILE table (#3910) (4 weeks ago) [mramezani95] * f15e2d0 - feat: support specific BP port info for show int (#3926) (4 weeks ago) [Chenyang Wang] * 5a59f19 - [multi-asic] Fix the 'config reload' flow in case when multiple golden_config.json files provided (#3895) (5 weeks ago) [Vadym Hlushko] * 82ec8f4 - fix show cmd for bgp (#3922) (5 weeks ago) [Liping Xu] * e0f9da4 - Skip checking offload flags for static routes/sids in route check and add check_sids (#3919) (5 weeks ago) [Changrong Wu] * 5ea861d - [copp]: Added CoPP show configuration commands (#3863) (6 weeks ago) [Ravi Minnikanti(Marvell)] * 9fd8c3c - [sfputil] Use host lane mask as part of rx-output enable/disable (#3911) (6 weeks ago) [mihirpat1] * 3e157a2 - Support reboot cause: Kernel Panic - Out of memory (#3918) (6 weeks ago) [byu343] * 49d36ff - [gcu]: Add marvell-teralynx platform to gcu field validator (#3881) (6 weeks ago) [Ravi Minnikanti(Marvell)] * 8415aee - [Mellanox] Collect sai.xml to sysdump (#3903) (6 weeks ago) [Sai Rama Mohan Reddy S] * 6e26c8d - [intfstat] Align output format between cached/non-cached scenarios (#3902) (6 weeks ago) [Yair Raviv] * 57d825e - Add version_202411_02 function (#3864) (6 weeks ago) [Ben Levi] * d5051cd - [Smartswitch][reboot] Addition of pre shutdown and post startup function calls (#3900) (7 weeks ago) [Gagan Punathil Ellath] * b3509b9 - Add CLI show commands to view bgp network, neighbors and summary on per-vrf basis (#3866) (7 weeks ago) [Navdha Jindal] * dfa51d3 - Upgrade portstat to support nonzero option and sort heterogeneous interfaces names (#3894) (7 weeks ago) [Changrong Wu] * ba255b6 - Issue #22407: ConfigReload fails when RADIUS statistics is enabled (#3860) (8 weeks ago) [Anders Linn] * 7116edf - Fix warm-reboot script so it can be run via reboot DBus service (#3872) (8 weeks ago) [jkmar] * 2f1c4e0 - config: Modify AAA config commands to use pass_db decorator (#3755) (8 weeks ago) [Anders Linn] * d6d866f - show command for icmp echo offload sessions (#3889) (8 weeks ago) [manamand2020] * 1b3498c - add TH5-512 hwsku into gcu support list (#3896) (8 weeks ago) [Dashuai Zhang] * b106a82 - Addition of prober_type in config and show commands for muxcable (#3884) (9 weeks ago) [harjotsinghpawra] * 733bdde - [smartswitch] Fix incorrect reboot status check and improve debug logging in reboot scripts (#3888) (9 weeks ago) [Vasundhara Volam] * 60110fa - feat: support namespace arg for show mac (#3873) (9 weeks ago) [Chenyang Wang] * aeba823 - feat: support namespace arg for show bfd (#3885) (9 weeks ago) [Chenyang Wang] ``` #### How I did it #### How to verify it #### Description for the changelog

Lawrence Lee and others added 14 commits December 12, 2019 10:48

[files]: add namespace synchronization target

504c632

* create systemd target to start/stop all containers in a specific namespace at once Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

[database docker]: add multi-instance support

8571864

* add multi-instance option to redis-cli * create database template service * change docker_image_ctl to use new redis sockets * increase database service timeout duration to 120s Signed-off-by: Lawrence Lee <t-lale@microsoft.com>

[files]: add multi-instance support for syncd

193f182

* add syncd service template * add instance number argument for syncd script

[files]: add multi-instance support to logging

aa44fda

* add rsyslog service template compatible with network namespaces * change rsyslog_config to use database instance unix socket Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit e871f11)

[files]: add namespace wrapper

f4844e8

* add wrapper for 'sudo ip netns exec' command Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 85745b7)

[platform/vs]: add memory to virtual switch

a13a8d8

* increase from 2GB to 8GB to prevent OOM crashes * necessary because multi-ASIC devices have increased memory requirements Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 845778e)

[build_templates]: add multi-instance support to teamd

9328339

* change teamd service file to template * force teamd to restart when swss does Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit 0297279)

[sonic-utilities]: update sonic-utilities

04b4887

* set submodule to point at newest commit Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit ebab66d)

[files]: change namespace name

d5c4a0d

* use 'asic' as namespace prefix instead of 'namespace' in accordance with PR sonic-net#3269 Signed-off-by: Lawrence Lee <t-lale@microsoft.com> (cherry picked from commit c1444e3)

Remove namespace docker related files.

83ca4de

[build_templates]: remove namespace service dependency

6e3b519

lguohan requested a review from qiluo-msft January 29, 2020 19:05

lguohan reviewed Jan 29, 2020

View reviewed changes

abdosi reviewed Jan 30, 2020

View reviewed changes

qiluo-msft self-assigned this Jan 31, 2020

jleveque suggested changes Jan 31, 2020

View reviewed changes

dockers/docker-database/base_image_files/redis-cli Outdated Show resolved Hide resolved

SuvarnaMeenakshi added 8 commits February 11, 2020 18:04

Merge remote-tracking branch 'remotes/origin/master' into sumeenak_pr_2

87220d9

[redis-cli/sonic-cfggen]: Add wrappers to talk to database instance in

81b024e

different namespaces.

[swss.sh]: Modified to use redis-cli wrapper to talk to multiple

1875e6b

database instances

[syncd.sh]: Updated to use redis-cli wrapper to talk to multiple

3363c0d

database instances.

[docker_img_ctl.j2]: Updated to use redis-cli/sonic-cfggen wrappers

95fb5f5

to talk to different database instances

[database service]: changes made to include global database service

382f13e

for multi-asic platform

[swss/syncd services]: Added dependency to topology.service for VS

c20bf43

platform

[interfaces-config.sh]: Made changes to support multi-asic platform

a76eb01

[syncd.servic]: Minor update in template file to make it concise.

fa25e1a

paulnice approved these changes Mar 19, 2020

View reviewed changes

jleveque approved these changes Mar 19, 2020

View reviewed changes

qiluo-msft approved these changes Mar 19, 2020

View reviewed changes

lguohan approved these changes Mar 20, 2020

View reviewed changes

yxieca approved these changes Mar 21, 2020

View reviewed changes

SuvarnaMeenakshi merged commit 4b8067e into sonic-net:master Mar 31, 2020

abdosi added the Request for 201911 Branch label Apr 2, 2020

abdosi pushed a commit that referenced this pull request Apr 2, 2020

Multi-ASIC implementation (#3888)

2e87a16

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

abdosi added the Included in 201911 Branch label Apr 2, 2020

nazariig mentioned this pull request Apr 3, 2020

[mellanox] SONiC fails to start on t0 #4367

Closed

abdosi added a commit that referenced this pull request Apr 3, 2020

Revert "Multi-ASIC implementation (#3888)"

249265a

This reverts commit 2e87a16.

SuvarnaMeenakshi mentioned this pull request Apr 5, 2020

[sonic-netns-exec]: use "$@" to reflects all positional parameters as they were set initially #4375

Merged

abdosi removed the Included in 201911 Branch label Apr 7, 2020

abdosi added the Included in 201911 Branch label Apr 15, 2020

abdosi pushed a commit that referenced this pull request Apr 15, 2020

Multi-ASIC implementation (#3888)

0099305

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Multi-ASIC implementation (sonic-net#3888)

aa616e8

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Revert "Multi-ASIC implementation (sonic-net#3888)"

485b9ff

This reverts commit 2e87a16.

tiantianlv pushed a commit to SONIC-DEV/sonic-buildimage that referenced this pull request Apr 24, 2020

Multi-ASIC implementation (sonic-net#3888)

eaa1d49

Changes made to support multi-asic platform. Added multi-instance support for swss, syncd, database, bgp, teamd and lldp.

SuvarnaMeenakshi deleted the pr_2_multiasic branch December 14, 2020 09:28

arlakshm mentioned this pull request Aug 4, 2021

[redis-cli][multiasic]redis-cli supports namespace #8243

Closed

		@@ -0,0 +1,13 @@
		[Unit]
		Description=Update rsyslog configuration

Conversation

SuvarnaMeenakshi commented Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lguohan commented Jan 29, 2020

Uh oh!

lguohan Jan 29, 2020

Choose a reason for hiding this comment

Uh oh!

SuvarnaMeenakshi Feb 26, 2020

Choose a reason for hiding this comment

Uh oh!

lguohan Jan 29, 2020

Choose a reason for hiding this comment

Uh oh!

SuvarnaMeenakshi Feb 24, 2020

Choose a reason for hiding this comment

Uh oh!

abdosi Jan 30, 2020

Choose a reason for hiding this comment

Uh oh!

SuvarnaMeenakshi Feb 26, 2020

Choose a reason for hiding this comment

Uh oh!

qiluo-msft commented Jan 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

SuvarnaMeenakshi commented Mar 19, 2020

Uh oh!

paulnice left a comment

Choose a reason for hiding this comment

Uh oh!

lguohan commented Mar 20, 2020

Uh oh!

SuvarnaMeenakshi commented Mar 20, 2020

Uh oh!

SuvarnaMeenakshi commented Mar 21, 2020

Uh oh!

lguohan commented Mar 24, 2020

Uh oh!

SuvarnaMeenakshi commented Mar 26, 2020

Uh oh!

SuvarnaMeenakshi commented Mar 31, 2020

Uh oh!

wendani commented May 9, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

SuvarnaMeenakshi commented Dec 12, 2019 •

edited

Loading

qiluo-msft commented Jan 31, 2020 •

edited

Loading