Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-1069

Expose Prometheus remote-write configuration

    XMLWordPrintable

Details

    • Remote Write
    • OBSDA-4Allow configure pushing time-series data to remote endpoints
    • Done
    • OBSDA-4 - Allow configure pushing time-series data to remote endpoints
    • 100
    • 100% 100%
    • 0

    Description

      Remote write allows to replicate time-series data to a remote location. This is important for several scenarios like you want to use "remote-write enabled" systems (e.g. InfluxDB) for long-term storage and historical analysis; as well as for aggregating metrics across multiple clusters.

      Currently, remote-write is in an experimental stage in Prometheus[1] but the chances are high that it will be stable some time this year. Furthermore, we are using remote-write pretty extensively already for Telemetry as well as ACM in the near future. With that in mind, we think that we are in a perfect spot to move what we already have[2] from dev preview to at least tech preview.

      Acceptance criteria

      • mTLS support (important for positioning Red Hat's Advanced Cluster Manager (ACM) as they will need it for pushing metrics from OpenShift clusters into their central management solution backed by Observatorium.
      • Default configurations coming from Red Hat (such as Telemetry and ACM) should not be overridden. ACM for example may inject their configuration automatically post installation (mechanism to be discussed).

      Non-goals

      • Configuration isolation for cluster and user workload monitoring ConfigMap to allow separating remote-write configuration per "tenant" or "user".
      • Remote write for Thanos Ruler (this isn't supported yet, see https://github.com/thanos-io/thanos/issues/1724).

      Open questions

      • Do we want to expose a different API to make configuring an allow list easier for everyone rather than exposing relabeling configuration directly? Reason is that we want to avoid validating "syntax" requests in a BZ or internal.

      Documentation

      • New section inside the configuration chapter that describes how to setup remote-write with an example on how it looks like for standard remote write implementation. For both CMO and UWM.
      • A small note about implications on setting up remote-write to the overall Prometheus cluster.
      • How to configure security/auth (e.g. (m)TLS).
      • The API.
      • Tuning.
      • Proxy configuration (if not supported, then we need a statement).

      Other resources

      [1] https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec - "If specified, the remote_write spec. This is an experimental feature, it may change in any upcoming release in a breaking way." The experimental flag was removed.

      Attachments

        Issue Links

          Activity

            People

              jfajersk@redhat.com Jan Fajerski
              cvogel1 Christian Heidenreich (Inactive)
              Junqi Zhao Junqi Zhao
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: