chore: pvc storage alert by percentage

[michaelact]

No description provided.

[skl]

I understand the intention here, but it seems very similar to the existing PV alerts. Per the originally stated problem:

Problem: The existing KubePersistentVolumeFillingUp alert only says "storage will be full in 4 days" but doesn't tell you how full it is right now. Even after 4 days, it still says 4 days. This makes it hard to know if you need to act immediately.

There are already warning and critical severity alerts for KubePersistentVolumeFillingUp. The critical version means "act immediately", in your case it sounds like only the warning version is firing.

The description on this warning alert currently says "Based on recent sampling, the PersistentVolume [...] is expected to fill up within four days. Currently {{ $value | humanizePercentage }} is available."

Perhaps it's always firing because you have some kind of sawtooth-wave which triggers the warning every time your rate of usage increases, maybe it's an indication that your PV is oscillating close to the 4-day prediction indicator?

Rather than creating a new alert, it sounds like maybe one or more of the following might apply:

1. The alert should be silenced
2. The capacity of the PV should be increased
3. The existing volumeFullPredictionSampleTime should be increased to make the alert less noisy
4. The alert should be modified so that it can be configurably less noisy

What do you think?

[github-actions]

This PR has been automatically marked as stale because it has not
had any activity in the past 30 days.

The next time this stale check runs, the stale label will be
removed if there is new activity. The issue will be closed in 7
days if there is no new activity.

Thank you for your contributions!