1. Proposed title of this feature request

Prometheus generating disk activity every two hours causing storage backend issues.

2. What is the nature and description of the request?

We're seeing Prometheus doing some type of disk activity every two hours on the hour on all of our clusters. We'd like to change that default setting so that all clusters aren't hitting our storage at the same time. Need help in finding where to make that config change. I see a knowledgebase article which says this is by design, but we'd like to stagger these if possible. [1][2]

3. Why does the customer need this? (List the business requirements here)

It appears to be impacting their storage clusters. They use Netapp Trident NFS as their PVC backing which serves multiple clusters and the Prometheus-k8s pods use Netapp Trident NFS PVCs for their data. It appears that this 2 hour interval job occurs at the exact time in every cluster and their hope is stagger this in each cluster such as: 

Those two hours for every cluster are midnight, 2:00AM, 4:00AM, etc... The question I've had is, can we change it so one cluster does midnight, 2:00AM, 4:00AM, etc... and another cluster does 12:15AM, 2:15AM, 4:15AM, etc... so they both aren't writing to storage at the same time? It's still a 2 hr default.

4. List any affected packages or components.

openshift-monitoring

[1] https://access.redhat.com/solutions/6960833
[2] https://prometheus.io/docs/prometheus/latest/storage/