Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61766

PrometheusRemoteWriteBehind alert firing when a remote-write queue drops all samples

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • MON Sprint 276, MON Sprint 277
    • 2
    • Done
    • Bug Fix
    • Hide
      Before this update, Prometheus Remote-Write was causing alerts due to dropped samples in relabeling configuration. As a consequence, the Prometheus Remote-Write Behind alert was active. With this release, the remote-write issue has been resolved, and dropped samples no longer trigger the alert. As a result, the Prometheus Remote Write Behind alert no longer becomes active when samples are dropped due to relabeling configuration.
      Show
      Before this update, Prometheus Remote-Write was causing alerts due to dropped samples in relabeling configuration. As a consequence, the Prometheus Remote-Write Behind alert was active. With this release, the remote-write issue has been resolved, and dropped samples no longer trigger the alert. As a result, the Prometheus Remote Write Behind alert no longer becomes active when samples are dropped due to relabeling configuration.
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-61706. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-61486. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-61207. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-56568. The following is the description of the original issue:
      —
      Description of problem:

      When Prometheus is configured to send samples to a remote-write endpoint but the relabeling configuration drops all samples then the PrometheusRemoteWriteBehind alert will become active.

      Version-Release number of selected component (if applicable):

          4.19 (should apply to all)

      How reproducible:

          Always

      Steps to Reproduce:

      (I'm using UWM as an example but it should work the same for platform monitoring)
    1. Deploy a sample application and configure metrics ingestion.
    2. Configure remote-write for UWM Prometheus but dropping all samples: 
data:
  config.yaml: |-
    prometheus:
      remoteWrite:
      - url: http://example.com/
        writeRelabelConfigs:
        - action: drop
          sourceLabels: [__name__]
     3. Check pending alerts in the OCP console

      Actual results:

      PrometheusRemoteWriteBehind shows as active.  

      Expected results:

      PrometheusRemoteWriteBehind shouldn't be active.  

      Additional info:

      The issue affects the upstream alerting rule.

              rh-ee-amrini Ayoub Mrini
              spasquie@redhat.com Simon Pasquier
              None
              None
              Tai Gao Tai Gao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: