Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-43690

Prometheus write_relabel_configs in remotewrite unable to drop metric in Grafana

XMLWordPrintable

    • Important
    • None
    • MON Sprint 261, MON Sprint 262
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      * With this release, the Cluster Monitoring Operator (CMO) has improved validation criteria. The CMO blocks cluster updates with configurations in `openshift-monitoring/cluster-monitoring-config` or `openshift-user-workload-monitoring/user-workload-monitoring-config` that include unsupported fields or misconfigurations. (link:https://issues.redhat.com/browse/OCPBUGS-43690[*OCPBUGS-43690*])
      ------
      To prepare for upcoming releases where CMO's configuration validation will be stricter, in this release, CMO will block cluster upgrades if the configurations in `openshift-monitoring/cluster-monitoring-config` or `openshift-user-workload-monitoring/user-workload-monitoring-config` fail to meet the stricter validation criteria. This is meant to encourage users to clean up their configurations by removing unsupported fields and correcting any misconfigurations. The goal is to establish a solid foundation for those future releases that will make it easier to identify configuration errors.


      (Is this really and Enhancement?)
      Show
      * With this release, the Cluster Monitoring Operator (CMO) has improved validation criteria. The CMO blocks cluster updates with configurations in `openshift-monitoring/cluster-monitoring-config` or `openshift-user-workload-monitoring/user-workload-monitoring-config` that include unsupported fields or misconfigurations. (link: https://issues.redhat.com/browse/OCPBUGS-43690 [* OCPBUGS-43690 *]) ------ To prepare for upcoming releases where CMO's configuration validation will be stricter, in this release, CMO will block cluster upgrades if the configurations in `openshift-monitoring/cluster-monitoring-config` or `openshift-user-workload-monitoring/user-workload-monitoring-config` fail to meet the stricter validation criteria. This is meant to encourage users to clean up their configurations by removing unsupported fields and correcting any misconfigurations. The goal is to establish a solid foundation for those future releases that will make it easier to identify configuration errors. (Is this really and Enhancement?)
    • Enhancement
    • Done

      This is a clone of issue OCPBUGS-42671. The following is the description of the original issue:

      Description of problem:

         Prometheus write_relabel_configs in remotewrite unable to drop metric in Grafana  

      Version-Release number of selected component (if applicable):

          

      How reproducible:

       Customer has tried both configurations to drop MQ metric with source_label(configuration 1) or without source_label(configuration 2) but it's not working.
      
      It seems to me that  drop configuration is not working properly and is buggy. 
      
      
      Configuration 1:
      
      ```
       remoteWrite:
              - url: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
                write_relabel_configs:
                - source_labels: ['__name__']
                  regex: 'ibmmq_qmgr_uptime'
                  action: 'drop'
                basicAuth:
                  username:
                    name: kubepromsecret
                    key: username
                  password:
                    name: kubepromsecret
                    key: password
      ```
      
      Configuration 2:
      ```
      remoteWrite:
              - url: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
                write_relabel_configs:
                - regex: 'ibmmq_qmgr_uptime'
                  action: 'drop'
                basicAuth:
                  username:
                    name: kubepromsecret
                    key: username
                  password:
                    name: kubepromsecret
                    key: password
      ```
      
      
      Customer wants to know what's the correct remote_write configuration to drop metric in Grafana ?
      
      Document links:
      https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write
      https://docs.openshift.com/container-platform/4.14/observability/monitoring/configuring-the-monitoring-stack.html#configuring-remote-write-storage_configuring-the-monitoring-stack
      https://docs.openshift.com/container-platform/4.14/observability/monitoring/configuring-the-monitoring-stack.html#creating-user-defined-workload-monitoring-configmap_configuring-the-monitoring-stack

      Steps to Reproduce:

          1.
          2.
          3.
          

      Actual results:

          prometheus remote_write configurations NOT droppping metric in Grafana   

      Expected results:

      prometheus  remote_write configurations should drop metric in Grafana    

      Additional info:

          

              rh-ee-amrini Ayoub Mrini
              openshift-crt-jira-prow OpenShift Prow Bot
              Junqi Zhao Junqi Zhao
              Eliska Romanova Eliska Romanova
              Jan Fajerski, Simon Pasquier
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: