Uploaded image for project: 'OpenShift Monitoring'
  1. OpenShift Monitoring
  2. MON-3369

Observing a rise in prometheus metrics while the cluster remains in an idle state

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Undefined Undefined
    • None
    • None
    • Prometheus
    • None
    • False
    • None
    • False
    • NEW
    • NEW
    • Hide
      1. An openshift-cluster with latest version (4.13.x)
      2. Values of these metrics: 

                max(alertmanager_http_request_duration_seconds_sum)

                sum(process_cpu_seconds_total{job="prometheus-k8s"})

                prometheus_tsdb_compaction_duration_seconds_count

           3. Cluster in Idle state. No workloads. Collecting the same metrics after 10 days.  

      Show
      An openshift-cluster with latest version (4.13.x) Values of these metrics:            max(alertmanager_http_request_duration_seconds_sum)           sum(process_cpu_seconds_total{job="prometheus-k8s"})           prometheus_tsdb_compaction_duration_seconds_count      3. Cluster in Idle state. No workloads. Collecting the same metrics after 10 days.  

      We have noticed an increase in the following metrics over a ten day span with the clusters being idle in both the architectures s390x and x86. 
      1. Sum of durations of Http requests made to the alert manager

      2. Total CPU time (in seconds) consumed by all processes associated with the "prometheus-k8s" job.
      3.  One metric which is used to measure and track the number of compaction operations performed by the Prometheus TSDB.

        1. s390x_1.jpg
          141 kB
          Mohammed Niha Tahoor
        2. s390x_2.jpg
          143 kB
          Mohammed Niha Tahoor
        3. s390x_3.jpg
          135 kB
          Mohammed Niha Tahoor
        4. x86_1.jpg
          147 kB
          Mohammed Niha Tahoor
        5. x86_2.jpg
          128 kB
          Mohammed Niha Tahoor
        6. x86_3.jpg
          138 kB
          Mohammed Niha Tahoor

              Unassigned Unassigned
              mtahoor@redhat.com Mohammed Niha Tahoor
              Axel Busch, Jitendra Singh
              Holger Wolf Holger Wolf
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: