metrics-exporter: add new metrics and alert to check for stale subvolumes #3570
Conversation
This commit introduces two new metrics- the total number of cephfs pv and the total number of subvolumes in a cephfilesystem. Signed-off-by: ShravaniVangur <[email protected]>
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ShravaniVangur The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
If total number of pvs and subvolumes do not match, the alert gets triggered. Signed-off-by: ShravaniVangur <[email protected]>
ShravaniVangur
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
| return nil, fmt.Errorf("no CephFilesystem CRs found") | ||
| } | ||
|
|
||
| fsName := cephFilesystems.Items[0].Name |
There was a problem hiding this comment.
Since the fsName is of the first filesystem, do we need to check if there is only one filesystem or a comment with reasoning for Items[0]
| rookClient: rookclient.NewForConfigOrDie(opts.Kubeconfig), | ||
| cephClusterNamespace: opts.AllowedNamespaces[0], | ||
| cephAuthNamespace: opts.CephAuthNamespace, | ||
| monitorConfig: cephMonitorConfig{}, |
There was a problem hiding this comment.
i think cephMonitorConfig is never initialised, we need to run initCeph for it to be initialised
| ocs_cephfs_pv_count != ocs_cephfs_subvolume_count and on() | ||
| (hour() == 0 and day_of_week() == 1) |
There was a problem hiding this comment.
do we want this alert to only run on Monday 0:00?
this might also not run, because the condition of hour()==0 will only be true for 60 minutes
but our requirements for the alert is the condition should be true for 6h
| subvolGroupCounts, err := c.runCephfsSubvolumeCountFn(c.monitorConfig, c.rookClient, c.cephClusterNamespace) | ||
| if err != nil { | ||
| klog.Errorf("failed to get CephFS subvolume counts during Replace: %v", err) | ||
| } else { | ||
| klog.Infof("CephFS subvolumegroup counts during Replace: %v", subvolGroupCounts) | ||
| c.CephFSSubvolumeCountMap = subvolGroupCounts |
There was a problem hiding this comment.
Collect function readsCephFSSubvolumeCountMap with RLock but we don't write with a lock acquired
3 participants
This pr introduces two new metrics- the total number of cephfs pv and the total number of subvolumes in a cephfilesystem.
This pr also introduces an alert which gets triggered when stale subvolumes are present.
Ref: https://issues.redhat.com/browse/RHSTOR-7531
//Todo: Addition of test files.