Implementation of a Monitoring Daemon for storage devices in SONiC switches#433
Merged
prgeor merged 31 commits intosonic-net:masterfrom May 31, 2024
Merged
Implementation of a Monitoring Daemon for storage devices in SONiC switches#433prgeor merged 31 commits intosonic-net:masterfrom
prgeor merged 31 commits intosonic-net:masterfrom
Conversation
ashwnsri
changed the title
This was referenced Feb 9, 2024
ashwnsri
requested review from
Junchao-Mellanox,
Staphylo,
keboliu and
prgeor
dgsudharsan
reviewed
May 22, 2024
dgsudharsan
requested changes
May 22, 2024
Junchao-Mellanox
requested changes
May 22, 2024
Merged
10 tasks
dgsudharsan
reviewed
May 23, 2024
Junchao-Mellanox
approved these changes
May 23, 2024
dgsudharsan
requested changes
May 23, 2024
dgsudharsan
approved these changes
May 24, 2024
added 15 commits
May 25, 2024 01:44
…ge*' to include all disk types
prgeor
reviewed
May 29, 2024
prgeor
reviewed
May 29, 2024
Contributor
Author
Added to the PR. |
dgsudharsan
requested changes
May 29, 2024
dgsudharsan
approved these changes
May 29, 2024
prgeor
reviewed
May 30, 2024
prgeor
reviewed
May 30, 2024
prgeor
reviewed
May 30, 2024
prgeor
reviewed
May 30, 2024
prgeor
reviewed
May 30, 2024
|
|
Contributor
Author
|
/azpw run |
Collaborator
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
prgeor
reviewed
May 30, 2024
sonic-stormond/scripts/stormond
Outdated
|
|
||
| STORAGEUTIL_LOAD_ERROR = 127 | ||
|
|
||
| log = syslogger.SysLogger(SYSLOG_IDENTIFIER) |
Collaborator
There was a problem hiding this comment.
@assrinivasan can we move this inside daemon calss?
prgeor
reviewed
May 30, 2024
|
|
||
| if value is None: self.log_warning("{}:{} value = None in StateDB".format(storage_device, field)) | ||
|
|
||
| self.statedb_storage_info_loaded = True |
Collaborator
There was a problem hiding this comment.
@assrinivasan what if the value is None, in that case we should fall back to .json on the disk
Contributor
Author
There was a problem hiding this comment.
Fixed this in latest. Also added a None check in the _load_fsio_rw_json function for None values. In this scenario, Both StateDB and JSON file have junk values, so it will be considered an init case.
prgeor
reviewed
May 30, 2024
sonic-stormond/scripts/stormond
Outdated
Comment on lines
+191
to
+200
| if self.statedb_storage_info_loaded == False and self.fsio_json_file_loaded == True: | ||
| self.use_fsio_json_baseline = True | ||
| self.use_statedb_baseline = False | ||
|
|
||
| # If stormond is coming back up after a daemon crash, storage information would be saved in the | ||
| # STATE_DB. In that scenario, we use the STATE_DB information as the SoT and reconcile the FSIO | ||
| # reads and writes values. | ||
| elif self.statedb_storage_info_loaded == True: | ||
| self.use_fsio_json_baseline = False | ||
| self.use_statedb_baseline = True |
Collaborator
There was a problem hiding this comment.
@assrinivasan can you make the logic more clear, i.e, if the stats are available in STATE_DB, then use that and as a fallback use .json values from the backup
Contributor
Author
|
/azpw run |
Collaborator
|
/AzurePipelines run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
prgeor
previously approved these changes
May 30, 2024
prgeor
approved these changes
May 31, 2024
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This commit adds a monitoring daemon for Storage device attributes on a device running SONiC.
SONiC Storage Monitoring Daemon HLD
Motivation and Context
Storage devices experience performance degradation over time on account of a variety of factors such as overall disk writes, bad-blocks management, lack of free space, sub-optimal operational temperature and good-old wear-and-tear which speaks to the overall health of the disk.
The goal of the Storage Monitoring Daemon (storagemond) is to provide meaningful metrics for the aforementioned issues and enable streaming telemetry for these attributes so that the required preventative measures are triggered in the eventuality of performance degradation.
How Has This Been Tested?
Has been manually tested on following platforms:
7050cx3.txt
S6100.txt
SN2700.txt
Additional Information (Optional)