Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61766

PrometheusRemoteWriteBehind alert firing when a remote-write queue drops all samples

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • MON Sprint 276, MON Sprint 277
    • 2
    • In Progress
    • Bug Fix
    • * With this release, PrometheusRemoteWriteBehind alerts will no longer fire if the remote endpoint has not yet received any data.
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-61706. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-61486. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-61207. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-56568. The following is the description of the original issue:

      Description of problem:

      When Prometheus is configured to send samples to a remote-write endpoint but the relabeling configuration drops all samples then the PrometheusRemoteWriteBehind alert will become active.

      Version-Release number of selected component (if applicable):

          4.19 (should apply to all)

      How reproducible:

          Always

      Steps to Reproduce:

      (I'm using UWM as an example but it should work the same for platform monitoring)
          1. Deploy a sample application and configure metrics ingestion.
          2. Configure remote-write for UWM Prometheus but dropping all samples: 
      data:
        config.yaml: |-
          prometheus:
            remoteWrite:
            - url: http://example.com/
              writeRelabelConfigs:
              - action: drop
                sourceLabels: [__name__]
           3. Check pending alerts in the OCP console
          

      Actual results:

      PrometheusRemoteWriteBehind shows as active.    

      Expected results:

      PrometheusRemoteWriteBehind shouldn't be active.    

      Additional info:

      The issue affects the upstream alerting rule.    

              rh-ee-amrini Ayoub Mrini
              spasquie@redhat.com Simon Pasquier
              None
              None
              Tai Gao Tai Gao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: