Orchagent send heartbeat during warm-reboot to prevent Orchagent stuck alert.

[liuh-80]

Orchangent send heartbeat during warm-reboot to prevent Orchagent stuck alert.

Why I did it

Orchangent will freese during warm-reboot, then supervisor-proc-exit-listener will generate false alert during warm reboot:
sonic-net/sonic-buildimage#16686

Work item tracking

• Microsoft ADO: 25295846

How I did it

Send heartbeat during warm-reboot freeze.

How to verify it

Pass all UT.
Manually verify issue fixed by check syslog.

Which release branch to backport (provide reason below if selected)

• 201811
• 201911
• 202006
• 202012
• 202106
• 202111
• 202205
• 202211
• 202305

Tested branch (Please provide the tested image version)

• SONiC.master-17060.400216-0d0a0dba4
• SONiC.202305-17081.401641-ec2aed854

Description for the changelog

Orchangent send heartbeat during warm-reboot to prevent Orchagent stuck alert.

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

[vaibhavhd]

I think this qualifies for a new UT, to check of heartbeats are still sent after Orchagent pause.

[liuh-80]

I think this qualifies for a new UT, to check of heartbeats are still sent after Orchagent pause.

Fixed, add heartbeat message check to warm reboot UT.

[liuh-80]

Trigger build again

[mssonicbld]

/AzurePipelines run Azure.sonic-swss

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dgsudharsan]

@liuh-80 Can you please address the pipeline failures?

[liuh-80]

@liuh-80 Can you please address the pipeline failures?

Sure, I'm still working on this PR.

[liuh-80]

The UT failed because supervisord not update process stdout to syslog immediately, it always delays few minutes, so test cause in this repo is very difficult to check the heartbeat signal.

Spend too much time on write UT in this repo, will add UT in sonic-mgmt repo to test heartbeat in warmreboot, which will be much easy.

[prsunny]

@qiluo-msft , @vaibhavhd to signoff

[dgsudharsan]

@liuh-80 This PR conflicts when I cherry-pick to 202305 branch. Can you please create a new PR for fixing it in 202305?

[dgsudharsan]

@prsunny @qiluo-msft @vaibhavhd Can you please sign off this PR?

[prsunny]

I think this qualifies for a new UT, to check of heartbeats are still sent after Orchagent pause.

Fixed, add heartbeat message check to warm reboot UT.

@liuh-80 , i dont see the UT changes. Can you check?

[liuh-80]

I think this qualifies for a new UT, to check of heartbeats are still sent after Orchagent pause.

Fixed, add heartbeat message check to warm reboot UT.

@liuh-80 , i dont see the UT changes. Can you check?

@prsunny , I try to add UT but found it's difficult to create a UT in this repo, because heartbeat message is send to systemd and sytemd have few minutes delay to write syslog.

So, my plan is to create test case in sonic-mgmt repo, will create PR in sonic-mgmt later.

Currently I manually verified change in this PR works.

[liuh-80]

UT created in sonic-mgmt repo: sonic-net/sonic-mgmt#10676
Will public UT for review after sonic-swss submodule update.

[StormLiangMS]

@liuh-80 could you help to update the test result with 202305?

[liuh-80]

@liuh-80 could you help to update the test result with 202305?

Description updated, tested on SONiC.202305-17081.401641-ec2aed854

[StormLiangMS]

@liuh-80 cherry pick conflict, could you file separate PR to 202305?

[liuh-80]

@liuh-80 cherry pick conflict, could you file separate PR to 202305?

202305 branch PR created: #2956