Add multi ASIC support for syslog rate limit feature by Junchao-Mellanox · Pull Request #3235 · sonic-net/sonic-utilities

Junchao-Mellanox · 2024-03-22T03:15:38Z

Build dependency sonic-net/sonic-buildimage#18459

What I did

Add multi ASIC support for syslog rate limit feature

How I did it

show syslog rate-limit-container add a new option namespace
config syslog rate-limit-container add a new option namespace
config syslog rate-limit-feature enable add a new option namespace and a new argument service_name
config syslog rate-limit-feature disable add a new option namespace and a new argument service_name

How to verify it

Manual test

Previous command output (if the output of a command-line utility has changed)

show syslog rate-limit-container help:

show syslog rate-limit-container --help
Usage: show syslog rate-limit-container [OPTIONS] <service_name>

  Show syslog rate limit configuration for containers

Options:
  -h, -?, --help  Show this message and exit.

config syslog rate-limit-container help

sudo config syslog rate-limit-container --help
Usage: config syslog rate-limit-container [OPTIONS] SERVICE_NAME

  Configure syslog rate limit for containers

Options:
  -i, --interval INTEGER RANGE  Configures syslog rate limit interval in
                                seconds for specified containers
  -b, --burst INTEGER RANGE     Configures syslog rate limit burst in number
                                of messages for specified containers
  -h, -?, --help                Show this message and exit.

config syslog rate-limit-feature enable help

sudo config syslog rate-limit-feature enable --help
Usage: config syslog rate-limit-feature enable [OPTIONS]

  Enable syslog rate limit feature

Options:
  -?, -h, --help  Show this message and exit.

config syslog rate-limit-feature disable help

sudo config syslog rate-limit-feature disable --help
Usage: config syslog rate-limit-feature disable [OPTIONS]

  Disable syslog rate limit feature

Options:
  -h, -?, --help  Show this message and exit.

New command output (if the output of a command-line utility has changed)

show syslog rate-limit-container help:

show syslog rate-limit-container --help
Usage: show syslog rate-limit-container [OPTIONS] <service_name>

  Show syslog rate limit configuration for containers

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

[new] show all config in multi ASIC platforms:

show syslog rate-limit-container 
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        500         40000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            50          50000
radv            300         20000
sflow           300         20000
snmp            300         20000


Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   500         40000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic1:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic2:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000


Namespace asic3:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   50          50000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000

[new] show global namespace configuration:

show syslog rate-limit-container -n default
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        500         40000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            50          50000
radv            300         20000
sflow           300         20000
snmp            300         20000

[new] show configuration for a given namespace:

show syslog rate-limit-container -n asic0
Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        50          50000
database   500         40000
gbsyncd    300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      300         20000

config syslog rate-limit-container help:

sudo config syslog rate-limit-container --help
Usage: config syslog rate-limit-container [OPTIONS] SERVICE_NAME

  Configure syslog rate limit for containers

Options:
  -i, --interval INTEGER RANGE    Configures syslog rate limit interval in
                                  seconds for specified containers
  -b, --burst INTEGER RANGE       Configures syslog rate limit burst in number
                                  of messages for specified containers
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

sudo config syslog rate-limit-feature enable help

sudo config syslog rate-limit-feature enable --help
Usage: config syslog rate-limit-feature enable [OPTIONS] [SERVICE_NAME]

  Enable syslog rate limit feature

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

sudo config syslog rate-limit-feature disable help

sudo config syslog rate-limit-feature disable --help
Usage: config syslog rate-limit-feature disable [OPTIONS] [SERVICE_NAME]

  Disable syslog rate limit feature

Options:
  -n, --namespace [asic0|asic1|asic2|asic3|default]
                                  Namespace name or all
  -h, -?, --help                  Show this message and exit.

Junchao-Mellanox · 2024-03-25T03:22:18Z

config/syslog.py

+                if feature_config[syslog_common.FEATURE_HAS_PER_ASIC_SCOPE].lower() == 'true':
+                    asic_count = multi_asic.get_num_asics()
+                    for i in range(asic_count):
+                        feature_list.append(f'{service_name}{i}')


Implement “get service name by namespace” in multi_asic.py

saiarcot895 · 2024-04-09T01:31:29Z

Looks like there are some test failures related to the changes here.

Also, I tested the current changes on a multi-ASIC KVM, and while it looks like containercfgd in a multi-instance container is getting the notification for the changed rate limit attributes, those changes aren't propagating into the rsyslog.conf file.

Junchao-Mellanox · 2024-04-09T01:38:27Z

Looks like there are some test failures related to the changes here.

Also, I tested the current changes on a multi-ASIC KVM, and while it looks like containercfgd in a multi-instance container is getting the notification for the changed rate limit attributes, those changes aren't propagating into the rsyslog.conf file.

Hi @saiarcot895 , there is a build dependency sonic-net/sonic-buildimage#18459, test will pass after that PR merging. I have tested the change on KVM, no issue found. If you see an issue, could you please provide your test step so that I can have a try? BTW, please make sure you have all 3 PRs in your test image:

saiarcot895 · 2024-04-09T02:00:26Z

Yes, I pulled in all 3 PRs in my test image, built a multi-ASIC image, loaded it, enabled the rate-limit feature, and changed the rate limiting:

admin@vlab-08:~$ sudo config syslog rate-limit-feature enable
Enabling syslog rate limit feature for database
Enabled syslog rate limit feature for database
Enabling syslog rate limit feature for dhcp_relay
dhcp_relay is not running, ignoring...
Enabling syslog rate limit feature for eventd
Enabled syslog rate limit feature for eventd
Enabling syslog rate limit feature for gnmi
Enabled syslog rate limit feature for gnmi
Enabling syslog rate limit feature for lldp
Enabled syslog rate limit feature for lldp
Enabling syslog rate limit feature for mgmt-framework
Enabled syslog rate limit feature for mgmt-framework
Enabling syslog rate limit feature for mux
mux is not running, ignoring...
Enabling syslog rate limit feature for nat
nat is not running, ignoring...
Enabling syslog rate limit feature for pmon
Enabled syslog rate limit feature for pmon
Enabling syslog rate limit feature for radv
Enabled syslog rate limit feature for radv
Enabling syslog rate limit feature for sflow
sflow is not running, ignoring...
Enabling syslog rate limit feature for snmp
Enabled syslog rate limit feature for snmp
Enabling syslog rate limit feature for bgp0
Enabled syslog rate limit feature for bgp0
Enabling syslog rate limit feature for database0
Enabled syslog rate limit feature for database0
Enabling syslog rate limit feature for gbsyncd0
Enabled syslog rate limit feature for gbsyncd0
Enabling syslog rate limit feature for lldp0
Enabled syslog rate limit feature for lldp0
Enabling syslog rate limit feature for macsec0
macsec0 is not running, ignoring...
Enabling syslog rate limit feature for swss0
Enabled syslog rate limit feature for swss0
Enabling syslog rate limit feature for syncd0
Enabled syslog rate limit feature for syncd0
Enabling syslog rate limit feature for teamd0
Enabled syslog rate limit feature for teamd0
Enabling syslog rate limit feature for bgp1
Enabled syslog rate limit feature for bgp1
Enabling syslog rate limit feature for database1
Enabled syslog rate limit feature for database1
Enabling syslog rate limit feature for gbsyncd1
Enabled syslog rate limit feature for gbsyncd1
Enabling syslog rate limit feature for lldp1
Enabled syslog rate limit feature for lldp1
Enabling syslog rate limit feature for macsec1
macsec1 is not running, ignoring...
Enabling syslog rate limit feature for swss1
Enabled syslog rate limit feature for swss1
Enabling syslog rate limit feature for syncd1
Enabled syslog rate limit feature for syncd1
Enabling syslog rate limit feature for teamd1
Enabled syslog rate limit feature for teamd1
Enabling syslog rate limit feature for bgp2
Enabled syslog rate limit feature for bgp2
Enabling syslog rate limit feature for database2
Enabled syslog rate limit feature for database2
Enabling syslog rate limit feature for gbsyncd2
Enabled syslog rate limit feature for gbsyncd2
Enabling syslog rate limit feature for lldp2
Enabled syslog rate limit feature for lldp2
Enabling syslog rate limit feature for macsec2
macsec2 is not running, ignoring...
Enabling syslog rate limit feature for swss2
Enabled syslog rate limit feature for swss2
Enabling syslog rate limit feature for syncd2
Enabled syslog rate limit feature for syncd2
Enabling syslog rate limit feature for teamd2
Enabled syslog rate limit feature for teamd2
Enabling syslog rate limit feature for bgp3
Enabled syslog rate limit feature for bgp3
Enabling syslog rate limit feature for database3
Enabled syslog rate limit feature for database3
Enabling syslog rate limit feature for gbsyncd3
Enabled syslog rate limit feature for gbsyncd3
Enabling syslog rate limit feature for lldp3
Enabled syslog rate limit feature for lldp3
Enabling syslog rate limit feature for macsec3
macsec3 is not running, ignoring...
Enabling syslog rate limit feature for swss3
Enabled syslog rate limit feature for swss3
Enabling syslog rate limit feature for syncd3
Enabled syslog rate limit feature for syncd3
Enabling syslog rate limit feature for teamd3
Enabled syslog rate limit feature for teamd3
admin@vlab-08:~$ sudo config syslog rate-limit-container -i 60 --burst 13000 teamd
admin@vlab-08:~$ show syslog rate-limit-container
SERVICE         INTERVAL    BURST
--------------  ----------  -------
database        300         20000
dhcp_relay      300         20000
eventd          300         20000
gnmi            300         20000
lldp            300         20000
mgmt-framework  300         20000
mux             300         20000
nat             300         20000
pmon            300         20000
radv            300         20000
sflow           300         20000
snmp            300         20000


Namespace asic0:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic1:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic2:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000


Namespace asic3:
SERVICE    INTERVAL    BURST
---------  ----------  -------
bgp        300         20000
database   300         20000
gbsyncd    300         20000
lldp       300         20000
macsec     300         20000
swss       300         20000
syncd      300         20000
teamd      60          13000
admin@vlab-08:~$ docker exec -it teamd1 cat /etc/rsyslog.conf | grep -i ratelimit
$SystemLogRateLimitInterval 300
$SystemLogRateLimitBurst 20000
admin@vlab-08:~$ sonic-db-cli -n asic0 CONFIG_DB HGETALL 'SYSLOG_CONFIG_FEATURE|teamd'
{'rate_limit_burst': '13000', 'rate_limit_interval': '60'}

Snippet from syslog:

Apr  9 01:59:53.475729 vlab-08 NOTICE teamd0#containercfgd[269]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.476077 vlab-08 NOTICE teamd1#containercfgd[345]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.477254 vlab-08 NOTICE teamd2#containercfgd[227]: Configure syslog rate limit interval=60, burst=13000
Apr  9 01:59:53.477758 vlab-08 NOTICE teamd3#containercfgd[227]: Configure syslog rate limit interval=60, burst=13000

Junchao-Mellanox · 2024-04-09T03:43:02Z

Hi @saiarcot895 , I have fixed it. The fix is in PR sonic-net/sonic-buildimage#18438

Junchao-Mellanox · 2024-04-15T08:45:04Z

/azpw run Azure.sonic-utilities

mssonicbld · 2024-04-15T08:45:06Z

/AzurePipelines run Azure.sonic-utilities

azure-pipelines · 2024-04-15T08:45:15Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik

i see new CLI commands added , should you add them to the command line reference md file as well?

Junchao-Mellanox · 2024-04-16T08:06:19Z

/azpw run Azure.sonic-utilities

mssonicbld · 2024-04-16T08:06:21Z

/AzurePipelines run Azure.sonic-utilities

azure-pipelines · 2024-04-16T08:06:31Z

Azure Pipelines successfully started running 1 pipeline(s).

Junchao-Mellanox · 2024-04-16T08:06:36Z

i see new CLI commands added , should you add them to the command line reference md file as well?

Will update it

Junchao-Mellanox · 2024-04-17T03:39:10Z

/azpw run Azure.sonic-utilities

mssonicbld · 2024-04-17T03:39:12Z

/AzurePipelines run Azure.sonic-utilities

azure-pipelines · 2024-04-17T03:39:21Z

Azure Pipelines successfully started running 1 pipeline(s).

liat-grozovik · 2024-04-17T08:38:08Z

@saiarcot895 @keboliu could you please help to review?

* Add multi ASIC support for syslog rate limit feature * Update command ref

Junchao-Mellanox · 2024-04-18T06:44:17Z

Cherry-pick PR: #3275

* Add multi ASIC support for syslog rate limit feature * Update command ref

Junchao-Mellanox marked this pull request as draft March 22, 2024 03:15

Junchao-Mellanox commented Mar 25, 2024

View reviewed changes

Junchao-Mellanox force-pushed the master-syslog-multiasic branch 4 times, most recently from 6296c7b to 64c1bfd Compare March 26, 2024 06:07

Junchao-Mellanox mentioned this pull request Apr 1, 2024

Add multi ASIC support for syslog rate limit feature sonic-net/SONiC#1648

Merged

Add multi ASIC support for syslog rate limit feature

7c48ef0

Junchao-Mellanox force-pushed the master-syslog-multiasic branch from 64c1bfd to 7c48ef0 Compare April 2, 2024 01:35

Junchao-Mellanox requested review from keboliu and saiarcot895 April 7, 2024 00:54

Junchao-Mellanox marked this pull request as ready for review April 7, 2024 00:54

Junchao-Mellanox added the Request for 202311 Branch label Apr 15, 2024

mssonicbld added the Cherry Pick Conflict_202311 label Apr 15, 2024

liat-grozovik reviewed Apr 16, 2024

View reviewed changes

Update command ref

50a2823

liat-grozovik approved these changes Apr 17, 2024

View reviewed changes

saiarcot895 approved these changes Apr 18, 2024

View reviewed changes

saiarcot895 merged commit 29d4e88 into sonic-net:master Apr 18, 2024

Junchao-Mellanox deleted the master-syslog-multiasic branch April 18, 2024 06:32

Junchao-Mellanox added a commit to Junchao-Mellanox/sonic-utilities that referenced this pull request Apr 18, 2024

Add multi ASIC support for syslog rate limit feature (sonic-net#3235)

5adb449

* Add multi ASIC support for syslog rate limit feature * Update command ref

Junchao-Mellanox mentioned this pull request Apr 18, 2024

[202311] Add multi ASIC support for syslog rate limit feature #3275

Closed

keboliu removed Request for 202311 Branch Cherry Pick Conflict_202311 labels Jun 5, 2024

arfeigin pushed a commit to arfeigin/sonic-utilities that referenced this pull request Jun 16, 2024

Add multi ASIC support for syslog rate limit feature (sonic-net#3235)

3a8673c

* Add multi ASIC support for syslog rate limit feature * Update command ref

nmoray pushed a commit to nmoray/sonic-utilities that referenced this pull request Jun 25, 2025

Add multi ASIC support for syslog rate limit feature (sonic-net#3235)

417f119

* Add multi ASIC support for syslog rate limit feature * Update command ref

yuazhe mentioned this pull request Nov 13, 2025

Liquid Cooling leakage detection in SONiC sonic-net/SONiC#2032

Merged

Conversation

Junchao-Mellanox commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Uh oh!

Junchao-Mellanox Mar 25, 2024

Choose a reason for hiding this comment

Uh oh!

saiarcot895 commented Apr 9, 2024

Uh oh!

Junchao-Mellanox commented Apr 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saiarcot895 commented Apr 9, 2024

Uh oh!

Junchao-Mellanox commented Apr 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Junchao-Mellanox commented Apr 15, 2024

Uh oh!

mssonicbld commented Apr 15, 2024

Uh oh!

azure-pipelines bot commented Apr 15, 2024

Uh oh!

liat-grozovik left a comment

Choose a reason for hiding this comment

Uh oh!

Junchao-Mellanox commented Apr 16, 2024

Uh oh!

mssonicbld commented Apr 16, 2024

Uh oh!

azure-pipelines bot commented Apr 16, 2024

Uh oh!

Junchao-Mellanox commented Apr 16, 2024

Uh oh!

Junchao-Mellanox commented Apr 17, 2024

Uh oh!

mssonicbld commented Apr 17, 2024

Uh oh!

azure-pipelines bot commented Apr 17, 2024

Uh oh!

liat-grozovik commented Apr 17, 2024

Uh oh!

Junchao-Mellanox commented Apr 18, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Junchao-Mellanox commented Mar 22, 2024 •

edited

Loading

Junchao-Mellanox commented Apr 9, 2024 •

edited

Loading

Junchao-Mellanox commented Apr 9, 2024 •

edited

Loading