Skip to content

Conversation

@weirdwiz
Copy link
Contributor

@weirdwiz weirdwiz commented Nov 13, 2025

PersistentVolumeUsage alerts were firing once per node for RWX volumes because kubelet on each node reports volume stats independently and Prometheus adds a node label during scraping. For RWX volumes mounted on N nodes, this created N identical alerts. The fix deduplicates across nodes while keeping only namespace/PVC labels.

image

PersistentVolumeUsage* alerts were firing once per node for RWX
volumes because kubelet on each node reports volume stats
independently and Prometheus adds a node label during scraping. For
RWX volumes mounted on N nodes, this created N identical alerts. The
fix deduplicates across nodes while keeping only namespace/PVC
labels.

Signed-off-by: Divyansh Kamboj <[email protected]>
@weirdwiz weirdwiz force-pushed the fix/deduplicate-rwx-pvc-alerts branch from f55834e to a81e488 Compare November 13, 2025 06:49
@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.21 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.20

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.19

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.18

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.20 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.19 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.17

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.16

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.18 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.17 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@weirdwiz
Copy link
Contributor Author

/cherry-pick release-4.15

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.16 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: once the present PR merges, I will cherry-pick it on top of release-4.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

@weirdwiz: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.20
/cherrypick release-4.19
/cherrypick release-4.18
/cherrypick release-4.17
/cherrypick release-4.16
/cherrypick release-4.15

In response to this:

/jira backport release-4.20,release-4.19,release-4.18,release-4.17,release-4.16,release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-cherrypick-robot

@openshift-ci-robot: once the present PR merges, I will cherry-pick it on top of release-4.15, release-4.16, release-4.17, release-4.18, release-4.19, release-4.20 in new PRs and assign them to you.

In response to this:

@weirdwiz: The following backport issues have been created:

Queuing cherrypicks to the requested branches to be created after this PR merges:
/cherrypick release-4.20
/cherrypick release-4.19
/cherrypick release-4.18
/cherrypick release-4.17
/cherrypick release-4.16
/cherrypick release-4.15

In response to this:

/jira backport release-4.20,release-4.19,release-4.18,release-4.17,release-4.16,release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@weirdwiz weirdwiz changed the title Deduplicate PVC usage alerts across nodes DFBUGS-4531: Deduplicate PVC usage alerts across nodes Nov 13, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid jira ticket of any type jira/invalid-bug Indicates that the referenced jira bug is invalid for the branch this PR is targeting labels Nov 13, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented Nov 13, 2025

@weirdwiz: This pull request references [Jira Issue DFBUGS-4531](https://issues.redhat.com//browse/DFBUGS-4531), which is invalid:

  • expected the bug to target the "odf-4.20" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

PersistentVolumeUsage alerts were firing once per node for RWX volumes because kubelet on each node reports volume stats independently and Prometheus adds a node label during scraping. For RWX volumes mounted on N nodes, this created N identical alerts. The fix deduplicates across nodes while keeping only namespace/PVC labels.

image

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 13, 2025
@Madhu-1
Copy link
Member

Madhu-1 commented Nov 13, 2025

/assign @iamniting @malayparida2000 for approval

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 13, 2025

@Madhu-1: GitHub didn't allow me to assign the following users: for, approval.

Note that only red-hat-storage members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @iamniting @malayparida2000 for approval

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@Madhu-1
Copy link
Member

Madhu-1 commented Nov 13, 2025

/retitle Deduplicate PVC usage alerts across nodes

@openshift-ci openshift-ci bot changed the title DFBUGS-4531: Deduplicate PVC usage alerts across nodes Deduplicate PVC usage alerts across nodes Nov 13, 2025
@openshift-ci-robot openshift-ci-robot removed jira/valid-reference Indicates that this PR references a valid jira ticket of any type jira/invalid-bug Indicates that the referenced jira bug is invalid for the branch this PR is targeting labels Nov 13, 2025
@openshift-ci-robot
Copy link

@weirdwiz: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

In response to this:

PersistentVolumeUsage alerts were firing once per node for RWX volumes because kubelet on each node reports volume stats independently and Prometheus adds a node label during scraping. For RWX volumes mounted on N nodes, this created N identical alerts. The fix deduplicates across nodes while keeping only namespace/PVC labels.

image

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 13, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: iamniting, Madhu-1, weirdwiz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 13, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit e639487 into red-hat-storage:main Nov 13, 2025
11 checks passed
@openshift-cherrypick-robot

@weirdwiz: new pull request created: #3578

In response to this:

/cherry-pick release-4.17

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: #3577 failed to apply on top of branch "release-4.16":

Applying: Deduplicate PVC usage alerts across nodes
Using index info to reconstruct a base tree...
M	controllers/storagecluster/prometheus/localcephrules.yaml
Falling back to patching base and 3-way merge...
Auto-merging controllers/storagecluster/prometheus/localcephrules.yaml
CONFLICT (content): Merge conflict in controllers/storagecluster/prometheus/localcephrules.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Deduplicate PVC usage alerts across nodes

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: #3577 failed to apply on top of branch "release-4.15":

Applying: Deduplicate PVC usage alerts across nodes
Using index info to reconstruct a base tree...
M	controllers/storagecluster/prometheus/localcephrules.yaml
Falling back to patching base and 3-way merge...
Auto-merging controllers/storagecluster/prometheus/localcephrules.yaml
CONFLICT (content): Merge conflict in controllers/storagecluster/prometheus/localcephrules.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 Deduplicate PVC usage alerts across nodes

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: new pull request created: #3579

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: new pull request created: #3580

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-cherrypick-robot

@weirdwiz: new pull request created: #3581

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants