Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-1069

Expose Prometheus remote-write configuration


    • Remote Write
    • Done
    • OBSDA-4 - Allow configure pushing time-series data to remote endpoints
    • OBSDA-4Allow configure pushing time-series data to remote endpoints
    • 0% To Do, 0% In Progress, 100% Done
    • 0

      Remote write allows to replicate time-series data to a remote location. This is important for several scenarios like you want to use "remote-write enabled" systems (e.g. InfluxDB) for long-term storage and historical analysis; as well as for aggregating metrics across multiple clusters.

      Currently, remote-write is in an experimental stage in Prometheus[1] but the chances are high that it will be stable some time this year. Furthermore, we are using remote-write pretty extensively already for Telemetry as well as ACM in the near future. With that in mind, we think that we are in a perfect spot to move what we already have[2] from dev preview to at least tech preview.

      Acceptance criteria

      • mTLS support (important for positioning Red Hat's Advanced Cluster Manager (ACM) as they will need it for pushing metrics from OpenShift clusters into their central management solution backed by Observatorium.
      • Default configurations coming from Red Hat (such as Telemetry and ACM) should not be overridden. ACM for example may inject their configuration automatically post installation (mechanism to be discussed).


      • Configuration isolation for cluster and user workload monitoring ConfigMap to allow separating remote-write configuration per "tenant" or "user".
      • Remote write for Thanos Ruler (this isn't supported yet, see https://github.com/thanos-io/thanos/issues/1724).

      Open questions

      • Do we want to expose a different API to make configuring an allow list easier for everyone rather than exposing relabeling configuration directly? Reason is that we want to avoid validating "syntax" requests in a BZ or internal.


      • New section inside the configuration chapter that describes how to setup remote-write with an example on how it looks like for standard remote write implementation. For both CMO and UWM.
      • A small note about implications on setting up remote-write to the overall Prometheus cluster.
      • How to configure security/auth (e.g. (m)TLS).
      • The API.
      • Tuning.
      • Proxy configuration (if not supported, then we need a statement).

      Other resources

      [1] https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec - "If specified, the remote_write spec. This is an experimental feature, it may change in any upcoming release in a breaking way." The experimental flag was removed.

            jfajersk@redhat.com Jan Fajerski
            cvogel1 Christian Heidenreich (Inactive)
            Junqi Zhao Junqi Zhao
            1 Vote for this issue
            15 Start watching this issue