[chassis][midplane] Add notification to Supervisor when LC is graceful reboot by mlok-nokia · Pull Request #3292 · sonic-net/sonic-utilities

mlok-nokia · 2024-04-26T02:09:19Z

What I did

Modify the "sudo reboot" script to notify the Supervisor card by creating/inserting CHASSIS_MODULE_REBOOT_INFO_TABLE|LINE-CARD#" entry to CHASSIS_STATE_DB when reboot command is issued on the Linecard. This provides the sufficient information to allow Supervisor to log a proper message to address issue sonic-net/sonic-buildimage#18540

How I did it

Add a new function linecard_reboot_notity_supervisor() to the reboot script. If this platform is a linecard in a chassis, call sonic-db-cli to add a "CHASSIS_MODULE_REBOOT_INFO_TABLE|LINE-CARD#" to the CHASSIS_STATE_DB. This provides the information to chassisd on Supervisor card to log a proper message.
This PRs requires the following 2 PRs to address issue sonic-net/sonic-buildimage#18540 :
sonic-net/sonic-buildimage#18805
sonic-net/sonic-platform-daemons#480
sonic-net/sonic-buildimage#18862

This PR is needed by branch 202205

How to verify it

Test expected log. Use the CLI command "sudo reboot" to reboot a linecard, then check the syslog on Supervisor. The below message is logged

Apr 25 19:44:40.818378 ixre-cpm-chassis7 WARNING pmon#chassisd: Expected: Module LINE-CARD0 lost midplane connectivity

Test unepxpected log. Using "sudo /sbin/reboot" or reboot a linecard with any crash method, then ccheck the syslog on Supervusor. The below message is logged.

Apr 25 19:50:22.549416 ixre-cpm-chassis7 WARNING pmon#chassisd: Unexpected: Module LINE-CARD0 lost midplane connectivity

Test the expexcted reboot with timeout case. Use the CLI command "sudo reboot" on linecard. and keep it down for more than 4 minutes. The below messages are logged.

Apr 25 01:25:53.877143 ixre-cpm-chassis7 WARNING sr_device_mgr: Unable to reach slot 1 (Linecard) via Midplane
Apr 25 01:25:58.402511 ixre-cpm-chassis7 WARNING pmon#chassisd: Module LINE-CARD0 went off-line!
Apr 25 01:26:01.658959 ixre-cpm-chassis7 WARNING pmon#chassisd: Expected: Module LINE-CARD0 lost midplane connectivity.
( 3 minutes after the first log)
Apr 25 01:29:10.259527 ixre-cpm-chassis7 WARNING pmon#chassisd: Unexpected: Module LINE-CARD0 midplane connectivity is not restored in 180 seconds

Previous command output (if the output of a command-line utility has changed)

NA

New command output (if the output of a command-line utility has changed)

NA

…l reboot Signed-off-by: mlok <marty.lok@nokia.com>

mlok-nokia · 2024-04-26T02:23:52Z

@deepak-singhal0408 @judyjoseph This PR is for an issue of logging lost midplane connectivity log. Total 3 PRs. Please review them. Thanks

abdosi · 2024-04-30T16:21:10Z

@bmridul please help review.

…ntry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

gechiang · 2024-05-03T22:28:41Z

@mlok-nokia ,
What is the dependency of this PR with "sonic-net/sonic-platform-daemons#480"?
If let's say we backport this to .msft repo 202205 branch but not the platorm-daemons PR (480), will there be any build issue or functionality issue? Reason I am asking this is because I don't think "sonic-net/sonic-platform-daemons#480" will be allowed to 202205 branch and since we don't have a .msft 202205 repo for this platform-deamons submodule, the complete bug fix will be incomplete for the community building with 202205... But we should be able tomake internal build with patch. Just want to make sure there are no negative impact to the rest of the community.
please confirm.
Thanks!

mlok-nokia · 2024-05-06T15:23:00Z

@mlok-nokia , What is the dependency of this PR with "sonic-net/sonic-platform-daemons#480"? If let's say we backport this to .msft repo 202205 branch but not the platorm-daemons PR (480), will there be any build issue or functionality issue? Reason I am asking this is because I don't think "sonic-net/sonic-platform-daemons#480" will be allowed to 202205 branch and since we don't have a .msft 202205 repo for this platform-deamons submodule, the complete bug fix will be incomplete for the community building with 202205... But we should be able tomake internal build with patch. Just want to make sure there are no negative impact to the rest of the community. please confirm. Thanks!

There should not be any functionality impact. PR #3292 just create a entry in CHASSIS_STATE_DB for platform-daemon PR sonic-net/sonic-platform-daemons#480 to use. If Platform-daemon is not in the branch, The data in DB will not be used.

judyjoseph

LGTM

judyjoseph · 2024-05-09T23:52:45Z

@kenneth-arista could you review as well

gechiang · 2024-05-15T17:49:10Z

MSFT ADO: 28074312
@StormLiangMS , @yxieca , Please help review/approve this BUG FIX for chassis for 202305 and 202311 branches.
Thanks!

…l reboot (sonic-net#3292) * [chassis][midplane] Add notification to Supervisor when LC is graceful reboot * Address review comment by adding log message when failed to create wentry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

mssonicbld · 2024-05-15T18:00:29Z

Cherry-pick PR to 202311: #3324

…l reboot (#3292) * [chassis][midplane] Add notification to Supervisor when LC is graceful reboot * Address review comment by adding log message when failed to create wentry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

… for Nokia-IXR7250E platform (#18862) This PR add the platform specified linecard_reboot_timeout value to the platform_evn.conf. It works PR sonic-net/sonic-platform-daemons#480 and sonic-net/sonic-utilities#3292 to address issue #18540 Signed-off-by: mlok <marty.lok@nokia.com>

gechiang · 2024-06-13T03:07:30Z

@StormLiangMS , no more backport allowed for 202305?? can you help review/approve/deny the backport request to 202305?
Thanks!

…l reboot (sonic-net#3292) * [chassis][midplane] Add notification to Supervisor when LC is graceful reboot * Address review comment by adding log message when failed to create wentry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

… for Nokia-IXR7250E platform (#18862) This PR add the platform specified linecard_reboot_timeout value to the platform_evn.conf. It works PR sonic-net/sonic-platform-daemons#480 and sonic-net/sonic-utilities#3292 to address issue #18540 Signed-off-by: mlok <marty.lok@nokia.com>

…l reboot (sonic-net#3292) * [chassis][midplane] Add notification to Supervisor when LC is graceful reboot * Address review comment by adding log message when failed to create wentry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

[chassis][midplane] Add notification to Supervisor when LC is gracefu…

c115038

…l reboot Signed-off-by: mlok <marty.lok@nokia.com>

abdosi requested a review from bmridul April 30, 2024 16:22

bmridul reviewed Apr 30, 2024

View reviewed changes

Comment thread scripts/reboot

Address review comment by adding log message when failed to create we…

dc557a9

…ntry in CHASSIS_STATE_DB Signed-off-by: mlok <marty.lok@nokia.com>

bmridul approved these changes May 2, 2024

View reviewed changes

deepak-singhal0408 approved these changes May 3, 2024

View reviewed changes

deepak-singhal0408 added Request for 202205 Branch Request for 202305 Branch labels May 3, 2024

gechiang added Chassis for 202205 Branch Request for 202311 Branch and removed Request for 202205 Branch labels May 3, 2024

judyjoseph approved these changes May 9, 2024

View reviewed changes

rlhui merged commit 547d5ee into sonic-net:master May 15, 2024

rlhui added the p0 label May 15, 2024

yxieca added the Approved for 202311 Branch label May 15, 2024

mssonicbld mentioned this pull request May 15, 2024

[action] [PR:3292] [chassis][midplane] Add notification to Supervisor when LC is graceful reboot (#3292) #3324

Merged

mssonicbld added the Created PR to 202311 Branch label May 15, 2024

mssonicbld added Included in 202311 Branch and removed Created PR to 202311 Branch labels May 15, 2024

gechiang mentioned this pull request May 22, 2024

[chassis][midplane] Add notification to Supervisor when LC is gracefu… Azure/sonic-utilities.msft#69

Merged

gechiang mentioned this pull request Jun 5, 2024

[Nokia][supervisor] add linecard_reboot_timeout value to platform_env… Azure/sonic-buildimage-msft#386

Merged

10 tasks

gechiang added the included in chassis for 202205 branch indicate that this PR got merged into the "chassis for 202205 branch" label Jun 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chassis][midplane] Add notification to Supervisor when LC is graceful reboot#3292

[chassis][midplane] Add notification to Supervisor when LC is graceful reboot#3292
rlhui merged 2 commits intosonic-net:masterfrom
mlok-nokia:midplane-connectivity-log

mlok-nokia commented Apr 26, 2024 •

edited

Loading

Uh oh!

mlok-nokia commented Apr 26, 2024

Uh oh!

abdosi commented Apr 30, 2024

Uh oh!

Uh oh!

gechiang commented May 3, 2024

Uh oh!

mlok-nokia commented May 6, 2024

Uh oh!

judyjoseph left a comment

Uh oh!

judyjoseph commented May 9, 2024

Uh oh!

gechiang commented May 15, 2024 •

edited

Loading

Uh oh!

mssonicbld commented May 15, 2024

Uh oh!

gechiang commented Jun 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Conversation

mlok-nokia commented Apr 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Uh oh!

mlok-nokia commented Apr 26, 2024

Uh oh!

abdosi commented Apr 30, 2024

Uh oh!

Uh oh!

gechiang commented May 3, 2024

Uh oh!

mlok-nokia commented May 6, 2024

Uh oh!

judyjoseph left a comment

Choose a reason for hiding this comment

Uh oh!

judyjoseph commented May 9, 2024

Uh oh!

gechiang commented May 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented May 15, 2024

Uh oh!

gechiang commented Jun 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mlok-nokia commented Apr 26, 2024 •

edited

Loading

gechiang commented May 15, 2024 •

edited

Loading