Skip to content

[cron.d] Add cron job to periodically clean-up core files #3449

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxieca merged 4 commits into from
Sep 13, 2019
Merged

[cron.d] Add cron job to periodically clean-up core files #3449

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
dae07e5
[cron.d] Create cron job to periodically clean-up core files
daall Sep 12, 2019
fe6728d
Update interval for running cron job
daall Sep 12, 2019
5222744
Respond to feedback
daall Sep 13, 2019
21a4ee0
Change syslog id
daall Sep 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions build_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,9 @@ build_number: ${BUILD_NUMBER:-0}
built_by: $USER@$BUILD_HOSTNAME
EOF

## Copy over clean-up script
sudo cp ./files/scripts/core_cleanup.py $FILESYSTEM_ROOT/usr/bin/core_cleanup

## Copy ASIC config checksum
python files/build_scripts/generate_asic_config_checksum.py
if [[ ! -f './asic_config_checksum' ]]; then
Expand Down
3 changes: 3 additions & 0 deletions files/image_config/cron.d/core_cleanup
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Attempts to clean up core files every 15 minutes
*/15 * * * * root /usr/bin/core_cleanup > /dev/null 2>&1

51 changes: 51 additions & 0 deletions files/scripts/core_cleanup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/usr/bin/env python

import syslog
import os

from collections import defaultdict
from datetime import datetime

SYSLOG_IDENTIFIER = 'core_cleanup_cronjob'
CORE_FILE_DIR = os.path.abspath('/var/core')
MAX_CORE_FILES = 4

def log_info(msg):
syslog.openlog(SYSLOG_IDENTIFIER)
syslog.syslog(syslog.LOG_INFO, msg)
syslog.closelog()

def log_error(msg):
syslog.openlog(SYSLOG_IDENTIFIER)
syslog.syslog(syslog.LOG_ERR, msg)
syslog.closelog()

def main():
if os.getuid() != 0:
log_error('Root required to clean up core files')
return

log_info('Cleaning up core files')
core_files = [f for f in os.listdir(CORE_FILE_DIR) if os.path.isfile(os.path.join(CORE_FILE_DIR, f))]

core_files_by_process = defaultdict(list)
for f in core_files:
Copy link
Copy Markdown
Contributor

@prsunny prsunny Sep 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a thought, should we have this as a generic infra so this can be used for cleanup other files also (old syslogs, swss.recs etc)?

Copy link
Copy Markdown
Contributor Author

@daall daall Sep 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good idea! I'll take a look at it.

Copy link
Copy Markdown
Contributor

@yxieca yxieca Sep 12, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great idea. Can we come back with another PR for that?

There are some specialty with core file handling, there are some persisted knowledge of core file name structure here. It is a bit hard to directly apply to log files.

But in general, I think we should have a more generic tool for other files.

Copy link
Copy Markdown
Contributor Author

@daall daall Sep 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I think we should put some more thought into a more generic way of doing this. One possible idea is to provide different functions to process different types of files, but let's come back it later.

process = f.split('.')[0]
curr_files = core_files_by_process[process]
curr_files.append(f)

if len(curr_files) > MAX_CORE_FILES:
curr_files.sort(reverse = True, key = lambda x: datetime.utcfromtimestamp(int(x.split('.')[1])))
oldest_core = curr_files[MAX_CORE_FILES]
log_info('Deleting {}'.format(oldest_core))
try:
os.remove(os.path.join(CORE_FILE_DIR, oldest_core))
except:
log_error('Unexpected error occured trying to delete {}'.format(oldest_core))
core_files_by_process[process] = curr_files[0:MAX_CORE_FILES]

log_info('Finished cleaning up core files')

if __name__ == '__main__':
main()