Proposed title of this feature request

Add scrape time jitter tolerations to UWM prometheus

What is the nature and description of the request?

Change the configuration of the UWM Prometheus instances to tolerate scrape time jitters.

Why does the customer need this? (List the business requirements)

Prometheus chunk compression relies on scrape times being accurately aligned to the scrape interval. Due to the nature of delta of delta encoding, a small delay from the configured scrape interval can cause tsdb data to occupy significantly more space.

We have observed a 50% difference in on disk tsdb storage for a replicated HA pair.

The downside is a reduction in sample accuracy and potential impact to derivatives of the time series. Allowing a jitter toleration will trade off improved chunk compression for reduced accuracy of derived data like the running average of a time series.

List any affected packages or components.

UWM Prometheus