Added code to start a new service to configure TSA as soon as Linecard comes up, start a timer to configure TSB and configure TSB when the timer expires by saksarav-nokia · Pull Request #206 · Azure/sonic-buildimage-msft

saksarav-nokia · 2024-01-04T16:47:56Z

Why I did it

The traffic loss is seen when the chassis or Linecard is rebooted

How I did it

Start a service startup_tsa_tsb.service when the IMM comes up
This service configures TSA and starts a timer (currently hardcoded to 900secs in device//startup-tsa-tsb.conf
When the timer expires, TSB is configured
Enhanced "sudo TSC" command output to indicate TSB pending with the remaining timer value and service name if the service is running
If the operator issues "sudo TSA" explicitly while the startup_tsa_tsb.service is running, service will be stopped.

How to verify it

Verified that the service is started only in IMM
Verified that the service runs only if device//startup-tsa-tsb.conf file exists
Verified that the TSA is configured and routes are not advertised to eBGP neighbors till TSB timer expires and TSB is configured.
"sudo TSC" shows the TSB Pending with remaining timer interval
Service is stopped when the operator issues "sudo TSA"

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

arlakshm

as comments

arlakshm · 2024-01-08T21:48:11Z

files/scripts/startup_tsa_tsb.py

+from threading import Timer
+import os.path
+
+def getPlatform():


use this function iso defining a new one
https://github.com/sonic-net/sonic-buildimage/blob/04d83c05aef84df531e2cb0c55c24dcd637f7e83/src/sonic-py-common/sonic_py_common/device_info.py#L89

Reused the existing function

arlakshm · 2024-01-08T21:51:31Z

files/scripts/startup_tsa_tsb.py

+    platform = (subprocess.check_output(['sonic-cfggen', '-d', '-v', platform_key.replace('"',"'")]).strip()).decode()
+    return platform
+
+def getNumAsics():


Resue this function https://github.com/sonic-net/sonic-buildimage/blob/04d83c05aef84df531e2cb0c55c24dcd637f7e83/src/sonic-py-common/sonic_py_common/multi_asic.py#L82?

Reused the existing function

arlakshm · 2024-01-08T21:52:56Z

files/scripts/startup_tsa_tsb.py

+         return line.split('=')[1].strip()
+    return 0
+
+def getTsbTimerInterval():


nit: please use snake case for functions names.

Modified the function names to use snake case

arlakshm · 2024-01-08T21:58:46Z

files/scripts/startup_tsa_tsb.py

+def getTsaConfig(asic_ns):
+    tsa_config = 'BGP_DEVICE_GLOBAL.STATE.tsa_enabled'
+    tsa_ena = (getSonicConfig(asic_ns, tsa_config)).decode()
+    print('{}: {} - CONFIG_DB.{} : {}'.format(__file__, asic_ns, tsa_config, tsa_ena))


can we please use logger iso print statements so that the message appear in syslog?

Changed the print to logger.log_info

arlakshm · 2024-01-08T21:59:35Z

files/scripts/startup_tsa_tsb.py

+def config_tsa():
+    tsa_ena = get_tsa_status()
+    if tsa_ena == True:
+       print("{}: Configuring TSA".format(__file__))


nit: please check the indentation in the file. It does not seem consistent

Fixed the indentation

arlakshm · 2024-01-08T22:17:12Z

files/scripts/startup_tsa_tsb.py

+         return line.split('=')[1].strip()
+    return 0
+
+def getSonicConfig(ns, config_name):


will this work for single asic linecards?

Modified the functions to work for both single asic and multi asic

arlakshm · 2024-01-08T22:19:02Z

files/build_templates/startup_tsa_tsb.service

+ConditionPathExists=!/etc/sonic/chassisdb.conf
+
+[Service]
+Environment="STARTED_BY_TSA_TSB_SERVICE=1"


after the service is stopped do we need to reset STARTED_BY_TSA_TSB_SERVICE

Reset the env variable

saksarav-nokia · 2024-01-09T20:53:32Z

@arlakshm , addressed all your comments. Please check

gechiang · 2024-01-10T04:21:55Z

@saksarav-nokia Why is this PR raised against this MSFT repo instead of on the public repo at: "https://github.com/sonic-net/sonic-buildimage"?
Is this a new feature that is not meant for public community?

saksarav-nokia · 2024-01-10T14:34:50Z

@saksarav-nokia Why is this PR raised against this MSFT repo instead of on the public repo at: "https://github.com/sonic-net/sonic-buildimage"? Is this a new feature that is not meant for public community?

@gechiang , It was suggested by MSFT in the meeting .

abdosi · 2024-01-10T16:58:13Z

files/build_templates/startup_tsa_tsb.service

+Description= STARTUP TSA-TSB SERVICE
+Requires=updategraph.service database.service
+After=updategraph.service database.service
+ConditionPathExists=!/etc/sonic/chassisdb.conf


what is need for this ?

updategraph dependency is to start the tsa-tsb service after config is loaded and chassisdb.conf file check is to start the service only in IMM.

abdosi · 2024-01-10T16:59:12Z

files/build_templates/startup_tsa_tsb.service

+[Unit]
+Description= STARTUP TSA-TSB SERVICE
+Requires=updategraph.service database.service
+After=updategraph.service database.service


Does it make sure we are running this service before swss/syncd/teamd/bgp services comes up ?

There is no dependency on these services right now. Ensure this service starts before bgp should be sufficient right?

Added "Before=bgp.service"

abdosi · 2024-01-10T17:00:58Z

files/build_templates/startup_tsa_tsb.service

+Environment="STARTED_BY_TSA_TSB_SERVICE=1"
+ExecStart=/usr/bin/python3 -u /usr/local/bin/startup_tsa_tsb.py  start
+ExecStop=/usr/bin/python3 -u /usr/local/bin/startup_tsa_tsb.py stop
+RemainAfterExit=yes


what is the behavior of this services in case of config reload and if the swss docker restart ?

config reload or swss restart doesn't restart the tsa service

gechiang · 2024-01-10T18:33:59Z

@saksarav-nokia Why is this PR raised against this MSFT repo instead of on the public repo at: "https://github.com/sonic-net/sonic-buildimage"? Is this a new feature that is not meant for public community?

@gechiang , It was suggested by MSFT in the meeting .

As discussed in the Chassis meeting today. It seems that this PR is supposed to be raised in this MSFT repo for the 202205 branch. Once it is approved and merged and well tested, this "feature" is also required in public master at which time please port it to "https://github.com/sonic-net/sonic-buildimage" master branch.
So the immediate action required for this PR is to move it to 202205 branch of this same msft repo.
Thanks!

…es up, start a timer to configure TSB and configure TSB when the timer expires Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>

1) Reused the existing Python functions 2) Modified the function names to use snake case 3) Changed the print to logger.log_info 4) Fixed the indentation 5) Modified it to work for both single and multi asic 6) Reset the env variable 7) When user issues TSB when the service is running, stop the service Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>

saksarav-nokia · 2024-01-10T19:36:02Z

@gechiang , i moved the PR to 202205

rlhui · 2024-01-10T21:50:32Z

please replace IMM with LC in the PR title so it's more generic

saksarav-nokia · 2024-01-11T00:20:42Z

please replace IMM with LC in the PR title so it's more generic

Changed the title

judyjoseph · 2024-01-11T06:46:39Z

files/scripts/startup_tsa_tsb.py

+    return tsa_ena
+
+def config_tsb():
+    logger.log_info("startup_tsa_tsb: Configuring TSB")


logging at line 70 and 80 could be consistent

judyjoseph · 2024-01-11T07:21:03Z

files/scripts/startup_tsa_tsb.py

+    return
+
+def stop_tsa_tsb():
+    reset_env_variables()


In case the user entered TSA/TSB command manually, shouldn't we explicitly stop/cancel timer thread which we started earlier in start_tsb_timer() ? Or does the systemd cleans it up during exit

Here we just reset the env variables only.

I will set the daemon flag for the Timer thread so that the thread will exit when the main thread is killed by the service stop

@judyjoseph , i tested with prctl and the threads started by the main thread are stopped when the service is stopped and main python script is killed with SIGTERM. So we don't need to explicitly stop the timer.

Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>

abdosi · 2024-01-22T20:20:30Z

cc @anamehra for viz.

…utomatically (#21565) #### Why I did it src/sonic-host-services ``` * 69788c2 - (HEAD -> master, origin/master, origin/HEAD) Fix: Set default values for kdump enhancement (#217) (3 days ago) [Muhammad Ali Hussnain] * d7e8021 - [chassis] make multi_asic_ns_to_host_fwd False for EXTERNAL_CLIENT acl service (#218) (4 days ago) [Arvindsrinivasan Lakshmi Narasimhan] * d925457 - Fix sudo command failed because root user password expired by password hardening feature issue (#215) (4 days ago) [Hua Liu] * 7e1b280 - reverted ssh_key to ssh_string (#216) (6 days ago) [Muhammad Ali Hussnain] * 741a9df - Handle NotImplementedError in determine-reboot-cause that will be thrown on VS Chassis platform (#211) (7 days ago) [Changrong Wu] * 47f6feb - check sonic-installer message and return proper error code for when image doesn't exist. (#210) (7 days ago) [Dawei Huang] * 1fe9a76 - Updated Key and Path (#209) (7 days ago) [Muhammad Ali Hussnain] * bb0a31c - Adding support for persistent storage and retrieval of DPU reboot-cause (#169) (8 days ago) [rameshraghupathy] * 5e08927 - [hostcfgd] Fix the state machine during eth0 default route check failure (#196) (2 weeks ago) [Vivek] * d2cc1a8 - Add ImageService.set_next_boot for GNOI Activate OS. (#207) (3 weeks ago) [Dawei Huang] * 9c49913 - register image_service and docker_service. (#208) (3 weeks ago) [Dawei Huang] * d88d8d0 - Implementation for ImageService.List (#206) (4 weeks ago) [Dawei Huang] * d7e4df5 - Enabled configuring the default number of kdumps in Linux. (#202) (4 weeks ago) [Mridul Bajpai] * ca9d329 - kdump-remote feature in hostcfgd (#166) (4 weeks ago) [Muhammad Ali Hussnain] ``` #### How I did it #### How to verify it #### Description for the changelog

…AD automatically (#1396) #### Why I did it src/sonic-utilities ``` * 7941e530 - (HEAD -> 202503, origin/202503) Merge pull request #206 from mssonicbld/sonicbld/202503-merge (23 hours ago) [mssonicbld] * 7e808a81 - Merge branch '202412' of https://github.com/Azure/sonic-utilities.msft into 202503 (23 hours ago) [Sonic Automation] * 992831ac - (origin/202412) fix show bgp cli on multiple asic device (#205) (24 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

…tomatically (#1983) #### Why I did it src/sonic-swss ``` * b7678f23 - (HEAD -> 202412, origin/202412) [action] [PR:4193] [countersyncd] fix otel actor log level (#204) (22 hours ago) [mssonicbld] * 803d2d78 - [action] [PR:4197] [countersyncd]: Modify the exit behavior of the main function (#206) (22 hours ago) [mssonicbld] ``` #### How I did it #### How to verify it #### Description for the changelog

saksarav-nokia requested a review from lguohan as a code owner January 4, 2024 16:47

judyjoseph requested review from abdosi, arlakshm, deepak-singhal0408 and judyjoseph January 8, 2024 19:43

arlakshm reviewed Jan 8, 2024

View reviewed changes

abdosi reviewed Jan 10, 2024

View reviewed changes

saksarav-nokia changed the base branch from master to 202205 January 10, 2024 19:15

saksarav-nokia requested review from a team, qiluo-msft and xumia as code owners January 10, 2024 19:15

saksarav-nokia added 2 commits January 10, 2024 14:26

Added code to start a new service to configure TSA as soon as IMM com…

d432d32

…es up, start a timer to configure TSB and configure TSB when the timer expires Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>

judyjoseph reviewed Jan 11, 2024

View reviewed changes

Added dependency to start the service before bgp service

d430346

Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>

gechiang approved these changes Feb 8, 2024

View reviewed changes

saksarav-nokia mentioned this pull request Mar 28, 2024

Linecard startup-TSA-TSB sonic-net/sonic-buildimage#18496

Merged

11 tasks

abdosi approved these changes Apr 5, 2024

View reviewed changes

gechiang merged commit 51a6a31 into Azure:202205 Apr 5, 2024

saksarav-nokia mentioned this pull request Apr 18, 2024

Fix for Bug in tsa-tsb script #338

Merged

10 tasks

Conversation

saksarav-nokia commented Jan 4, 2024 • edited by judyjoseph Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

A picture of a cute animal (not mandatory but encouraged)

Uh oh!

arlakshm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saksarav-nokia commented Jan 9, 2024

Uh oh!

gechiang commented Jan 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saksarav-nokia commented Jan 10, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abdosi Jan 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gechiang commented Jan 10, 2024

Uh oh!

saksarav-nokia commented Jan 10, 2024

Uh oh!

rlhui commented Jan 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

saksarav-nokia commented Jan 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

judyjoseph Jan 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

saksarav-nokia commented Jan 4, 2024 •

edited by judyjoseph

Loading

gechiang commented Jan 10, 2024 •

edited

Loading

abdosi Jan 10, 2024 •

edited

Loading

rlhui commented Jan 10, 2024 •

edited

Loading

judyjoseph Jan 11, 2024 •

edited

Loading