[consultancy] Create an external HA Monitoring instance for centralizing multiple-clusters data

Overview:

• In OpenShift Container Native Virtualization QE team we are frequently deploying clusters to run tests on, validate bug fixes, and more.
• OpenShift Container Platform includes a pre-configured, pre-installed, and self-updating monitoring stack that provides monitoring for core platform components. ([1])
• Currently, we are not persisting data that has been collected by Prometheus and the data's life cycle is as the pod's life-cycle.
• We would like to start persisting clusters data that has been collected by Prometheus.
• We would also like to take the current default stack and scale it horizontally with centralized Prometheus cluster.

The plan was:

• A sidecar will be added to each cluster that has been deployed by us in-order to store the data on an object storage (such as S3).
  • Filip Petkovski Has told me on #forum-monitoring channel that it's not possible.

• The central cluster will be connected to each cluster to query fresh data and to an object store for querying older data.  ([2] - #38)

Some Questions:

• Is it possible for us to store our monitoring data on OpenShift Monitoring team's Central Observatorium (Telemeter) cluster?
• Is the data being persisted on this central cluster automatically?
• What is the best use-case for persisting the data in case it's not being persisted automatically on this cluster?
  Why can't we modify the already deployed sidecar or deploy a new one so it will send blocks to S3?

Sources:

[1] Understanding the monitoring stack

[2] Thanos: Scaling out Prometheus to a Global, Durable Metric System; Red Hat Edition