[config] Enable/disable container monitoring when starting/stopping the services.#1471
Merged
yozhao101 merged 8 commits intosonic-net:masterfrom Mar 3, 2021
Merged
Conversation
stopping services and monitor it again after restarting services when ran the commands `sudo config reload`, `sudo config load` and `sudo config load_minigraph`. Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
jleveque
suggested changes
Mar 2, 2021
jleveque
suggested changes
Mar 2, 2021
Contributor
jleveque
left a comment
jleveque
left a comment
There was a problem hiding this comment.
PR title is deceptive. This change doesn't completely disable Monit, it only disables the monitoring of containers. Please update accordingly.
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
yozhao101
changed the title
Contributor
Author
Great suggestion, Updated! |
jleveque
suggested changes
Mar 2, 2021
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
jleveque
previously approved these changes
Mar 2, 2021
Signed-off-by: Yong Zhao <yozhao@microsoft.com>
jleveque
approved these changes
Mar 3, 2021
Contributor
Author
|
@jleveque I will create a separate PR to update the submodule. |
Contributor
|
@yozhao101, @jleveque this change cannot be cherry-picked to 202012 cleanly, Is there a dependency needs to be cherry-picked? I think this PR is depended on this one: #1199. |
Contributor
|
This commit has been reverted from the master branch, so I am removing the "Request for 202012 branch" label. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Yong Zhao yozhao@microsoft.com
What I did
When we ran the command
sudo config load,sudo config reloadorsudo config load_minigraph, the containersswss, snmp, lldp, teamd, syncd, snmp, bgp, radv, pmon, dhcp_relay, telemetryandrestapiwould be stopped and then restarted. The scriptcontainer_checkerran by Monit will generate false alerting messages into syslog to indicate some containers were not running during such stopping and restarting process. So this PR aims to prevent Monit from generating false alarm messages.How I did it
Before stopping services, we disable Monit to monitor the running status of containers. After restarting services, we enable Monit to monitor the running status of containers again.
How to verify it
I deliberately reduce the monitoring interval of Monit from 60 seconds to 10 seconds to ensure the alerting messages from the script
container_checkerwas generated duringsudo config reload, sudo config load and sudo config load_minigraph. After this change was added into_stop_services(...)and_restart_services(...), I checked that the alerting messages fromcontainer_checkerdid not appear in the syslog.I verified this change on the device
str-a7050-acs-3.Previous command output (if the output of a command-line utility has changed)
New command output (if the output of a command-line utility has changed)