Skip to content

[monit] Restart rsyslog service if rsyslogd consumes > 800 MB memory#3117

Merged
jleveque merged 1 commit intosonic-net:masterfrom
jleveque:rsyslog_monit
Jul 4, 2019
Merged

[monit] Restart rsyslog service if rsyslogd consumes > 800 MB memory#3117
jleveque merged 1 commit intosonic-net:masterfrom
jleveque:rsyslog_monit

Conversation

@jleveque
Copy link
Contributor

@jleveque jleveque commented Jul 3, 2019

Configure monit to monitor the resident memory consumption of rsyslogd. If memory usage is > 800 MB for 5 out of 10 checks (2-minute cycle interval, so 10 out of 20 minutes), restart the rsyslog service, because rsyslogd is most likely leaking memory. This is the same change that was committed to the 201803 branch here: #2963.

I tested an alternative solution using systemd (which in turn uses memory cgroups, which are now supported and enabled by default in Stretch). However, now that we have configured panic_on_oom = 2 (link), this will cause ALL OOM situations (even within memory cgroups) to trigger kernel panic. This defeats the purpose, as this solution is meant to prevent OOM situations, so I have decided to utilize monit for this purpose in the master branch (Stretch) as well as the 201803 (Jessie) branch.

@jleveque jleveque merged commit f14354f into sonic-net:master Jul 4, 2019
@jleveque jleveque deleted the rsyslog_monit branch July 4, 2019 01:21
@yxieca
Copy link
Contributor

yxieca commented Jul 10, 2019

Separate commit has been made to 201811 branch

@jleveque
Copy link
Contributor Author

@yxieca: You had temporarily cherry-picked from the 201803 branch until we had a solution on the master branch. You should probably replace it with this commit, even though they are identical for all intents and purposes.

@yxieca
Copy link
Contributor

yxieca commented Jul 10, 2019

Thanks Joe for keeping me honest! :-)

mssonicbld added a commit that referenced this pull request May 29, 2024
…lly (#19120)

#### Why I did it
src/sonic-swss
```
* 835e576 - (HEAD -> master, origin/master, origin/HEAD) [tests] mark suppress-fib-pending VS test as expected to fail (6 hours ago) [Stepan Blyshchak]
* 353ab92 - [subnet_decap] Add subnet decap (#3117) (10 hours ago) [Longxiang Lyu]
```
#### How I did it
#### How to verify it
#### Description for the changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants