Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30315

PrometheusMissingRuleEvaluations firing intermittently only for "kube-apiserver.rules" group

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.12
    • Monitoring
    • None
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Alert PrometheusMissingRuleEvaluations is firing intermittently only for "kube-apiserver.rules" group which is defined in "kube-apiserver-slos-extended" PrometheusRule (in openshift-kube-apiserver project).
      The alert gets resolved within 15 minutes of firing.
      
      Below warning is streamed repetitively every time before the alert fires:
      ~~~
      2024-02-09T13:34:48.214853072Z ts=2024-02-09T13:34:48.214Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: watch of *v1.Pod ended with: very short watch: github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:543: Unexpected watch close - watch lasted less than a second and no items received"
      ~~~

      Version-Release number of selected component (if applicable):

      RHOCP 4.12.45

      How reproducible:

      NA

      Steps to Reproduce:

          1. Not sure how to reproduce it
          2.
          3.
          

      Actual results:

      The alert PrometheusMissingRuleEvaluations intermittently for "kube-apiserver.rules" alert group.

      Expected results:

      The alert should not fire at all.

      Additional info:

      There are 304 prometheusrules defined in the cluster.
      There are no resource limits defined on prometheus pods. CPU utilization is around 3vCPU for 1 pod and 2vCPU for other pod.
      Interestingly, the doesn't stream for any other prometheusrule.

            janantha@redhat.com Jayapriya Pai
            rhn-support-dgautam Dhruv Gautam
            Junqi Zhao Junqi Zhao
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: