Skip to content

Fixed determine/process reboot-cause service dependency#17462

Merged
yxieca merged 1 commit intosonic-net:202205from
anamehra:anamehra/reboot_cause_202205
Dec 11, 2023
Merged

Fixed determine/process reboot-cause service dependency#17462
yxieca merged 1 commit intosonic-net:202205from
anamehra:anamehra/reboot_cause_202205

Conversation

@anamehra
Copy link
Copy Markdown
Contributor

@anamehra anamehra commented Dec 8, 2023

Why I did it

Fixes #16990 for 202305/202205 branch

Note: This PR is for 202305 and 202205. For master, a new PR will be raised with a new field (Uphold=) provided by debian bookworm to handle the dependency failure restartability of the processes.

  1. determine-reboot-cause and process-reboot-cause service does not start If the database service fails to restart in the first attempt. Even if the Database service succeeds in the next attempt, these reboot-cause services do not start.

  2. The process-reboot-cause service also does not restart if the docker or database service restarts, which leads to an empty reboot-cause history

  3. deploy-mg from sonic-mgmt also triggers the docker service restart. The restart of the docker service caused the issue stated in 2 above. The docker restart also triggers determine-reboot-cause to restart which creates an additional reboot-cause file in history and modifies the last reboot-cause.

This PR fixes these issues by making both processes start again when dependency meets after dependency failure, making both processes restart when the database service restarts, and preventing duplicate processing of the last reboot reason.

Work item tracking
  • Microsoft ADO 25892856

How I did it

  1. Modified systemd unit files to make determine-reboot-cause and process-reboot-cause services restartable when the database service restarts.
  2. On the restart, the determine-reboot-cause service should not recreate a new reboot-cause entry in the database. Added check for first start or restart to skip entry for restart case.

How to verify it

On single asic pizza box:

  1. Installed the image and check reboot-cause history
  2. restart database service and verify that determine-reboot-cause and process-reboot-cause services also restart. Verify that reboot-cause shows correct data and no new entry is created for restart.

On Chassis:

  1. Installed the image and check reboot-cause history
  2. restart the database service and verify that determine-reboot-cause and process-reboot-cause services also restart. Verify that reboot-cause shows correct data and no new entry is created for restart.
  3. Reboot LC. On Supervicor, stop database-chassis service.
    Let database service on LC fail the first time. determine-reboot-cause and process-reboot-cause would fail to start due to dependency failure
    start database-chassis on Supervisor. Database service on LC should now start successfully.
    Verify determine-reboot-cause and process-reboot-cause also starts
    Verify show reboot-cause history output

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: anamehra <anamehra@cisco.com>
@gechiang
Copy link
Copy Markdown
Collaborator

@yxieca This is the cherry-pick PR for the one (#17406) that you already approved for 202305 but requested a direct PR as it could not be cherry-picked cleanly. Please help merge it.

Thanks!

@yxieca yxieca merged commit e754b3b into sonic-net:202205 Dec 11, 2023
@anamehra anamehra deleted the anamehra/reboot_cause_202205 branch November 24, 2024 16:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants