Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-3369

Observing a rise in prometheus metrics while the cluster remains in an idle state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • None
    • Prometheus
    • None
    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      1. An openshift-cluster with latest version (4.13.x)
      2. Values of these metrics: 

                max(alertmanager_http_request_duration_seconds_sum)

                sum(process_cpu_seconds_total{job="prometheus-k8s"})

                prometheus_tsdb_compaction_duration_seconds_count

           3. Cluster in Idle state. No workloads. Collecting the same metrics after 10 days.  

      Show
      An openshift-cluster with latest version (4.13.x) Values of these metrics:            max(alertmanager_http_request_duration_seconds_sum)           sum(process_cpu_seconds_total{job="prometheus-k8s"})           prometheus_tsdb_compaction_duration_seconds_count      3. Cluster in Idle state. No workloads. Collecting the same metrics after 10 days.  

      We have noticed an increase in the following metrics over a ten day span with the clusters being idle in both the architectures s390x and x86. 
      1. Sum of durations of Http requests made to the alert manager

      2. Total CPU time (in seconds) consumed by all processes associated with the "prometheus-k8s" job.
      3.  One metric which is used to measure and track the number of compaction operations performed by the Prometheus TSDB.

        1. s390x_1.jpg
          s390x_1.jpg
          141 kB
        2. s390x_2.jpg
          s390x_2.jpg
          143 kB
        3. s390x_3.jpg
          s390x_3.jpg
          135 kB
        4. x86_1.jpg
          x86_1.jpg
          147 kB
        5. x86_2.jpg
          x86_2.jpg
          128 kB
        6. x86_3.jpg
          x86_3.jpg
          138 kB

              Unassigned Unassigned
              mtahoor@redhat.com Mohammed Niha Tahoor
              Axel Busch, Jitendra Singh
              Holger Wolf Holger Wolf
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: