Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-42671

Prometheus write_relabel_configs in remotewrite unable to drop metric in Grafana

XMLWordPrintable

    • Important
    • None
    • MON Sprint 260, MON Sprint 261
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, CMO wouldn't mark configurations in 'openshift-monitoring/cluster-monitoring-config' or 'openshift-user-workload-monitoring/user-workload-monitoring-config' as invalid for unknown (e.g. no longer supported) or duplicated fields. With this release, stricter validation helps identify such errors.
      Show
      Previously, CMO wouldn't mark configurations in 'openshift-monitoring/cluster-monitoring-config' or 'openshift-user-workload-monitoring/user-workload-monitoring-config' as invalid for unknown (e.g. no longer supported) or duplicated fields. With this release, stricter validation helps identify such errors.
    • Bug Fix
    • In Progress

      Description of problem:

         Prometheus write_relabel_configs in remotewrite unable to drop metric in Grafana  

      Version-Release number of selected component (if applicable):

          

      How reproducible:

       Customer has tried both configurations to drop MQ metric with source_label(configuration 1) or without source_label(configuration 2) but it's not working.
      
      It seems to me that  drop configuration is not working properly and is buggy. 
      
      
      Configuration 1:
      
      ```
       remoteWrite:
              - url: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
                write_relabel_configs:
                - source_labels: ['__name__']
                  regex: 'ibmmq_qmgr_uptime'
                  action: 'drop'
                basicAuth:
                  username:
                    name: kubepromsecret
                    key: username
                  password:
                    name: kubepromsecret
                    key: password
      ```
      
      Configuration 2:
      ```
      remoteWrite:
              - url: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
                write_relabel_configs:
                - regex: 'ibmmq_qmgr_uptime'
                  action: 'drop'
                basicAuth:
                  username:
                    name: kubepromsecret
                    key: username
                  password:
                    name: kubepromsecret
                    key: password
      ```
      
      
      Customer wants to know what's the correct remote_write configuration to drop metric in Grafana ?
      
      Document links:
      https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
      https://docs.openshift.com/container-platform/4.14/observability/monitoring/configuring-the-monitoring-stack.html#configuring-remote-write-storage_configuring-the-monitoring-stack
      https://docs.openshift.com/container-platform/4.14/observability/monitoring/configuring-the-monitoring-stack.html#creating-user-defined-workload-monitoring-configmap_configuring-the-monitoring-stack

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          prometheus remote_write configurations NOT droppping metric in Grafana   

      Expected results:

      prometheus  remote_write configurations should drop metric in Grafana    

      Additional info:

          

              rh-ee-amrini Ayoub Mrini
              sasakshi@redhat.com Sakshi sakshi
              Junqi Zhao Junqi Zhao
              Eliska Romanova Eliska Romanova
              Jan Fajerski, Simon Pasquier
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: