Prometheus will always run compaction at predefined times of day. This hold true when running multiple instances, which can cause high load spikes in the underlying storage.

A jitter for compaction scheduling would be a good addition to the tsdb. This is more complex then it appears. Today the blocks that are the output of compaction are aligned on full hour marks. This property is an assumption in many places of the prometheus code. So offsetting when compaction runs needs to still produce blocks that align on the full hour.

https://github.com/prometheus/prometheus/issues/8278