Add orchagent heart beat message for watchdog. by liuh-80 · Pull Request #2737 · sonic-net/sonic-swss

liuh-80 · 2023-04-17T05:25:28Z

What I did
Improve orch agent: output heartbeat message to systemd.

Why I did it
Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it.

How I verified it
Pass all UT.
Manually validate the heartbeat message works correctly.

Details if related
Another inprogress PR will add watchdog for this heartbeat message:
sonic-net/sonic-buildimage#14686

sonic-mgmt UT PR: sonic-net/sonic-mgmt#8306

orchagent/orchdaemon.cpp

qiluo-msft · 2023-05-19T07:33:55Z

orchagent/orchdaemon.cpp


+void OrchDaemon::heartBeat(std::chrono::time_point<std::chrono::high_resolution_clock> tcurrent)
+{
+    static auto tlast = std::chrono::high_resolution_clock::now();


static

You are assuming OrchDaemon has only single instance in the process. To be super safe, you can use a static member variable instead of a static function variable. #Closed

Fixed, change to a static member variable.

Sorry for misleading.

To be super safe, you can use a ~~static~~ member variable instead of a static function variable.

Fixed, change to none static member.

qiluo-msft · 2023-05-30T22:01:20Z

orchagent/orchdaemon.cpp

 #define DEFAULT_MAX_BULK_SIZE 1000
 size_t gMaxBulkSize = DEFAULT_MAX_BULK_SIZE;

+std::chrono::time_point<std::chrono::high_resolution_clock> OrchDaemon::m_lastHeartBeat = std::chrono::high_resolution_clock::now();


m_lastHeartBeat

std::chrono::time_pointstd::chrono::high_resolution_clock OrchDaemon::m_lastHeartBeat = std::chrono::high_resolution_clock::now();

If not static, should it be initialized in ctor? #Closed

Fixed, initialized in ctor

…ave issue. (#14686) This PR depends on sonic-net/sonic-swss#2737 merge first. **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (#15429) Add watchdog mechanism to swss service and generate alert when swss have issue. **Work item tracking** Microsoft ADO (number only): 16578912 **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually test process_monitoring/test_critical_process_monitoring.py can pass. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

**What I did** Improve orch agent: output heartbeat message to systemd. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually validate the heartbeat message works correctly. **Details if related** Another inprogress PR will add watchdog for this heartbeat message: sonic-net/sonic-buildimage#14686 sonic-mgmt UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (sonic-net#14686) This PR depends on sonic-net/sonic-swss#2737 merge first. **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

…ave issue. (sonic-net#15429) Add watchdog mechanism to swss service and generate alert when swss have issue. **Work item tracking** Microsoft ADO (number only): 16578912 **What I did** Add orchagent watchdog to monitor and alert orchagent stuck issue. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually test process_monitoring/test_critical_process_monitoring.py can pass. Add new UT sonic-net/sonic-mgmt#8306 to check watchdog works correctly. Manually test, after pause orchagent with 'kill -STOP <pid>', check there are warning message exist in log: Apr 28 23:36:41.504923 vlab-01 ERR swss#supervisor-proc-watchdog-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes). **Details if related** Heartbeat message PR: sonic-net/sonic-swss#2737 UT PR: sonic-net/sonic-mgmt#8306

**What I did** Improve orch agent: output heartbeat message to systemd. **Why I did it** Currently SONiC monit system only monit orchagent process exist or not. If orchagent process stuck and stop processing, current monit can't find and report it. **How I verified it** Pass all UT. Manually validate the heartbeat message works correctly. **Details if related** Another inprogress PR will add watchdog for this heartbeat message: sonic-net/sonic-buildimage#14686 sonic-mgmt UT PR: sonic-net/sonic-mgmt#8306

Add orchagent heartbeat message

7b973dc

liuh-80 changed the title ~~Add orchagent heart beat message~~ [POC] Add orchagent heart beat message Apr 17, 2023

liuh-80 added 5 commits April 17, 2023 07:55

Change to use supervisord message

f2be481

Update event message

dd87f93

Merge remote-tracking branch 'origin' into dev/liuh/add-heart-beat

392de22

Improve heartbeat message

ac63cf6

Improve code

0c928c8

liuh-80 requested a review from qiluo-msft April 28, 2023 10:16

liuh-80 changed the title ~~[POC] Add orchagent heart beat message~~ Add orchagent heart beat message for watchdog. Apr 28, 2023

liuh-80 marked this pull request as ready for review April 28, 2023 10:18

liuh-80 requested a review from prsunny as a code owner April 28, 2023 10:18

liuh-80 mentioned this pull request Apr 28, 2023

Add watchdog mechanism to swss service and generate alert when swss have issue. sonic-net/sonic-buildimage#14686

Merged

qiluo-msft reviewed May 11, 2023

View reviewed changes

orchagent/orchdaemon.cpp Outdated Show resolved Hide resolved

qiluo-msft reviewed May 12, 2023

View reviewed changes

orchagent/orchdaemon.cpp Outdated Show resolved Hide resolved

Imporve code by PR comments

b24ecb3

qiluo-msft reviewed May 19, 2023

View reviewed changes

liuh-80 added 2 commits May 19, 2023 08:16

Improve code by PR comments

12a3524

Fix code

7f17fd4

qiluo-msft reviewed May 30, 2023

View reviewed changes

liuh-80 force-pushed the dev/liuh/add-heart-beat branch from 4af1876 to 7f17fd4 Compare May 31, 2023 00:07

qiluo-msft approved these changes Jun 5, 2023

View reviewed changes

liuh-80 merged commit 99a2a26 into sonic-net:master Jun 6, 2023

liuh-80 mentioned this pull request Jun 12, 2023

Add watchdog mechanism to swss service and generate alert when swss have issue. sonic-net/sonic-buildimage#15429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add orchagent heart beat message for watchdog.#2737

Add orchagent heart beat message for watchdog.#2737
liuh-80 merged 9 commits intosonic-net:masterfrom
liuh-80:dev/liuh/add-heart-beat

liuh-80 commented Apr 17, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

qiluo-msft May 19, 2023 •

edited

Loading

Uh oh!

liuh-80 May 22, 2023 •

edited

Loading

Uh oh!

qiluo-msft May 30, 2023

Uh oh!

liuh-80 May 31, 2023

Uh oh!

qiluo-msft May 30, 2023 •

edited

Loading

Uh oh!

liuh-80 May 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

liuh-80 commented Apr 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qiluo-msft May 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liuh-80 May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qiluo-msft May 30, 2023

Choose a reason for hiding this comment

Uh oh!

liuh-80 May 31, 2023

Choose a reason for hiding this comment

Uh oh!

qiluo-msft May 30, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

std::chrono::time_pointstd::chrono::high_resolution_clock OrchDaemon::m_lastHeartBeat = std::chrono::high_resolution_clock::now();

Uh oh!

liuh-80 May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

liuh-80 commented Apr 17, 2023 •

edited

Loading

qiluo-msft May 19, 2023 •

edited

Loading

liuh-80 May 22, 2023 •

edited

Loading

qiluo-msft May 30, 2023 •

edited

Loading