[chassis][midplane] Modify the chassisd to log expected/unexpected midplane connectivity messages by mlok-nokia · Pull Request #480 · sonic-net/sonic-platform-daemons

mlok-nokia · 2024-04-26T01:42:17Z

Modified the SUP chassisd check_midplane_reachability() function to use the CHASSIS_MODULE_REBOOT_INFO_TABLE data (which is set by Linecard "sudo reboot" command) log expected or unexpected module lost midplane connectivity. This address issue sonic-net/sonic-buildimage#18540

Description

Add a new method is_module_reboot_expected() to check if CHASSIS_MODULE_REBOOT_INFO_TABLE|LINECARD# entry exists in CHASSIS_STATE_DB when a linecard is not reachable from SUP. If entry exists, it is expected reboot. check_midplane_reachability() will log "pmon#chassisd: Expected: Module LINE-CARD1 lost midplane connectivity". If entry doesn't exist, it will log "pmon#chassisd: Unexpected: Module LINE-CARD1 lost midplane connectivity". The CHASSIS_MODULE_REBOOT_INFO_TABLE|LINECARD# entry created and insert by linecard "sudo reboot" command by PR. It means that Users issue a linecard reboot, "lost midplane connectivity" is expected. Otherwise, such a linecard crash or missing heartbeat reboot, etc is unexpected.
Add new method module_reboot_set_time() and is_module_reboot_system_up_expired() to check if an expected reboot of linecard is not able to be up and detected by SUP in 3 minutes, check_midplane_reachabikity() will log "pmon#chassisd: Unexpected: Module LINE-CARD1 lost midplane connectivity". This provides the log message to the monitoring tool to take any further action.
This PR is required and associated with the following PRs
PR sonic-net/sonic-buildimage#18805
sonic-net/sonic-utilities#3292
#480
sonic-net/sonic-buildimage#18862

Motivation and Context

This provides a proper log message whether a module "lost midplane connectivity" is expected or not. This provides an efficient information log to the monitoring tool to take any further action. Fixes sonic-net/sonic-buildimage#18540

How Has This Been Tested?

This PR requires PRhttps://github.com/sonic-net/sonic-utilities/pull/3292 and to work with

Test expected log. Use the CLI command "sudo reboot" to reboot a linecard, then check the syslog on Supervisor. The below message is logged

Apr 25 19:44:40.818378 ixre-cpm-chassis7 WARNING pmon#chassisd: Expected: Module LINE-CARD0 lost midplane connectivity

Test unepxpected log. Using "sudo /sbin/reboot" or reboot a linecard with any crash method, then ccheck the syslog on Supervusor. The below message is logged.

Apr 25 19:50:22.549416 ixre-cpm-chassis7 WARNING pmon#chassisd: Unexpected: Module LINE-CARD0 lost midplane connectivity

Test the expexcted reboot with timeout case. Use the CLI command "sudo reboot" on linecard. and keep it down for more than 4 minutes. The below messages are logged.

Apr 25 01:25:53.877143 ixre-cpm-chassis7 WARNING sr_device_mgr: Unable to reach slot 1 (Linecard) via Midplane
Apr 25 01:25:58.402511 ixre-cpm-chassis7 WARNING pmon#chassisd: Module LINE-CARD0 went off-line!
Apr 25 01:26:01.658959 ixre-cpm-chassis7 WARNING pmon#chassisd: Expected: Module LINE-CARD0 lost midplane connectivity.
( 3 minutes after the first log)
Apr 25 01:29:10.259527 ixre-cpm-chassis7 WARNING pmon#chassisd: Unexpected: Module LINE-CARD0 midplane connectivity is not restored in 180 seconds

Additional Information (Optional)

This PR needs to be back ported to branchs:
[x] 202205

mlok-nokia · 2024-04-26T02:23:31Z

@deepak-singhal0408 @judyjoseph This PR is for an issue of logging lost midplane connectivity log. Total 3 PRs. Please review them. Thanks

bmridul · 2024-04-30T17:34:19Z

Can you provide details (schema) on Chassis Module Reboot Info table which is introduced here.
A small write up/enhancement to PMON Chassisd HLD will also be useful.

bmridul · 2024-04-30T18:12:03Z

It is not clear why the Chassis module reboot info entry needs to be removed from platform specific code. Isn't this handled entirely in sonic common code.

https://github.com/sonic-net/sonic-buildimage/blob/9ad556336d7e8216daaf6bd8a769532434b2f7e4/device/nokia/x86_64-nokia_ixr7250e_36x400g-r0/platform_reboot#L16

sonic-chassisd/scripts/chassisd

mlok-nokia · 2024-04-30T20:05:05Z

It is not clear why the Chassis module reboot info entry needs to be removed from platform specific code. Isn't this handled entirely in sonic common code.

https://github.com/sonic-net/sonic-buildimage/blob/9ad556336d7e8216daaf6bd8a769532434b2f7e4/device/nokia/x86_64-nokia_ixr7250e_36x400g-r0/platform_reboot#L16

On Nokia platform, one of the unpexpect reboot (missing heartbeart reboot) is calling the "sudo reboot". Since "sudo reboot" creates the expected CHASSIS_MODULE_REBOOT_INFO_TABLE entry, we need to remove it for this case. This is platform specified behaviors.

mlok-nokia · 2024-04-30T20:15:46Z

Can you provide details (schema) on Chassis Module Reboot Info table which is introduced here. A small write up/enhancement to PMON Chassisd HLD will also be useful.

The CHASSIS_MODULE_REBOOT_INFO_TABLE defined as below:
keys: "CHASSIS_MODULE_REBOOT_INFO_TABLE|"
value: "reboot" : "expected"

Example:
{
"CHASSIS_MODULE_REBOOT_INFO_TABLE|LINE-CARD1": {
"expireat": 1714507996.2318723,
"ttl": -0.001,
"type": "hash",
"value": {
"reboot": "expected"
}
}
}

deepak-singhal0408 · 2024-05-01T20:55:15Z

@mlok-nokia, could you please also add UT case?

…dplane connectivity messages Signed-off-by: mlok <[email protected]>

Add mechanism to get the linecard_reboot_timeout value from platform_env.conf file. This provides capabilitiy to different platform can have a different timeout value

deepak-singhal0408

LGTM.

deepak-singhal0408 · 2024-05-03T17:48:56Z

@bmridul @amulyan7 could you please have a look at the latest change share any further comments. Thanks!

mlok-nokia · 2024-05-03T17:51:15Z

@mlok-nokia, could you please also add UT case?

UT has been added

judyjoseph

LGTM, do we need to define this new table here : https://github.com/sonic-net/sonic-swss-common/blob/master/common/schema.h#L440

judyjoseph · 2024-05-09T23:52:38Z

@kenneth-arista could you review as well

deepak-singhal0408 · 2024-05-22T06:20:36Z

MSFT ADO: 28164958

… for Nokia-IXR7250E platform (#18862) This PR add the platform specified linecard_reboot_timeout value to the platform_evn.conf. It works PR sonic-net/sonic-platform-daemons#480 and sonic-net/sonic-utilities#3292 to address issue #18540 Signed-off-by: mlok <[email protected]>

mlok-nokia mentioned this pull request Apr 26, 2024

[chassis][midplane] Add notification to Supervisor when LC is graceful reboot sonic-net/sonic-utilities#3292

Merged

bmridul reviewed Apr 30, 2024

View reviewed changes

sonic-chassisd/scripts/chassisd Outdated Show resolved Hide resolved

sonic-chassisd/scripts/chassisd Outdated Show resolved Hide resolved

amulyan7 reviewed Apr 30, 2024

View reviewed changes

sonic-chassisd/scripts/chassisd Outdated Show resolved Hide resolved

amulyan7 reviewed Apr 30, 2024

View reviewed changes

sonic-chassisd/scripts/chassisd Outdated Show resolved Hide resolved

mlok-nokia force-pushed the midplane-module-connectivity-log branch 2 times, most recently from d38ebe6 to 386748a Compare April 30, 2024 19:56

mlok-nokia closed this Apr 30, 2024

mlok-nokia reopened this Apr 30, 2024

mlok-nokia added 2 commits May 3, 2024 09:45

[chassis][midplane] Modify the chassisd to log expected/unexpected mi…

e93f5b8

…dplane connectivity messages Signed-off-by: mlok <[email protected]>

Address review comments

8a214aa

mlok-nokia force-pushed the midplane-module-connectivity-log branch from 386748a to 918461f Compare May 3, 2024 13:48

mlok-nokia added 2 commits May 3, 2024 09:49

Address review comments

918461f

Add mechanism to get the linecard_reboot_timeout value from platform_env.conf file. This provides capabilitiy to different platform can have a different timeout value

Add UT test linecard reboot

29f675c

mlok-nokia mentioned this pull request May 3, 2024

[Nokia][supervisor] add linecard_reboot_timeout value to platform_env.conf for Nokia-IXR7250E platform sonic-net/sonic-buildimage#18862

Merged

11 tasks

deepak-singhal0408 approved these changes May 3, 2024

View reviewed changes

deepak-singhal0408 added Request for 202205 Branch Request for 202305 Branch labels May 3, 2024

bmridul approved these changes May 3, 2024

View reviewed changes

gechiang added the Request for 202311 Branch label May 3, 2024

gechiang mentioned this pull request May 3, 2024

[Nokia][chassis] modify Nokia-IXR7250E-36x400G platform specified reboot to allow SUP to log expected/unexpected midplane/module connectivity msg sonic-net/sonic-buildimage#18805

Merged

11 tasks

judyjoseph approved these changes May 9, 2024

View reviewed changes

rlhui merged commit 88bf8ec into sonic-net:master May 15, 2024

gechiang mentioned this pull request Jun 5, 2024

[Nokia][supervisor] add linecard_reboot_timeout value to platform_env… Azure/sonic-buildimage-msft#386

Merged

10 tasks

mlok-nokia mentioned this pull request Aug 7, 2024

[chassis][pmon][chassid] Enhance the chassid module on-line or off-line log messages with physical slot number #530

Merged

Conversation

mlok-nokia commented Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

How Has This Been Tested?

Additional Information (Optional)

Uh oh!

mlok-nokia commented Apr 26, 2024

Uh oh!

bmridul commented Apr 30, 2024

Uh oh!

bmridul commented Apr 30, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mlok-nokia commented Apr 30, 2024

Uh oh!

mlok-nokia commented Apr 30, 2024

Uh oh!

deepak-singhal0408 commented May 1, 2024

Uh oh!

deepak-singhal0408 left a comment

Choose a reason for hiding this comment

Uh oh!

deepak-singhal0408 commented May 3, 2024

Uh oh!

mlok-nokia commented May 3, 2024

Uh oh!

judyjoseph left a comment

Choose a reason for hiding this comment

Uh oh!

judyjoseph commented May 9, 2024

Uh oh!

deepak-singhal0408 commented May 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mlok-nokia commented Apr 26, 2024 •

edited

Loading