Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61706

PrometheusRemoteWriteBehind alert firing when a remote-write queue drops all samples

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • 4.18.0
    • 4.16, 4.17, 4.18, 4.19, 4.20
    • Monitoring
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • MON Sprint 276
    • 1
    • In Progress
    • Bug Fix
    • Hide
      Before this update, the `PrometheusRemoteWriteBehind` alert fired if the remote endpoint never received any data. With this release, the `PrometheusRemoteWriteBehind`alert no longer fires if the remote endpoint has not yet received any data. (link:https://issues.redhat.com/browse/OCPBUGS-61706[OCPBUGS-61706])
      Show
      Before this update, the `PrometheusRemoteWriteBehind` alert fired if the remote endpoint never received any data. With this release, the `PrometheusRemoteWriteBehind`alert no longer fires if the remote endpoint has not yet received any data. (link: https://issues.redhat.com/browse/OCPBUGS-61706 [ OCPBUGS-61706 ])
    • None
    • None
    • None
    • None

      This is a clone of issue OCPBUGS-61486. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-61207. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-56568. The following is the description of the original issue:
      —
      Description of problem:

      When Prometheus is configured to send samples to a remote-write endpoint but the relabeling configuration drops all samples then the PrometheusRemoteWriteBehind alert will become active.

      Version-Release number of selected component (if applicable):

          4.19 (should apply to all)

      How reproducible:

          Always

      Steps to Reproduce:

      (I'm using UWM as an example but it should work the same for platform monitoring)
    1. Deploy a sample application and configure metrics ingestion.
    2. Configure remote-write for UWM Prometheus but dropping all samples: 
data:
  config.yaml: |-
    prometheus:
      remoteWrite:
      - url: http://example.com/
        writeRelabelConfigs:
        - action: drop
          sourceLabels: [__name__]
     3. Check pending alerts in the OCP console

      Actual results:

      PrometheusRemoteWriteBehind shows as active.  

      Expected results:

      PrometheusRemoteWriteBehind shouldn't be active.  

      Additional info:

      The issue affects the upstream alerting rule.

              jfajersk@redhat.com Jan Fajerski
              spasquie@redhat.com Simon Pasquier
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: