Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14075

Prometheus does not recognize termination of pods

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • 4.10
    • Monitoring
    • Moderate
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      The prometheus-operator (Metrics) does not recognize when a pod is terminated, hence it keeps sending the metrics from terminated pods last received. In our receiving monitoring stack (ELK), these data lead to errors in the resulting metrics.

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Open-shift cluster in the version mentioned above with activated prometheus. Run several Pods with software that cases metrics and configure them so that they are scraped by the prometheus, that is configured to send the metrics via remote-write to an external monitoring-stack (e.g.: elastic). Terminate some pods. 

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      Prometheus keeps sending the data last received from terminated pods

      Expected results:

      Prometheus should not send metric-data from terminated pods and should not-send received metric-data more than once.

      Additional info:

       

              spasquie@redhat.com Simon Pasquier
              wolfgang.hoerleinsberger@itsv.at Wolfgang Hörleinsberger (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: