Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-3916

expr for SDNPodNotReady is wrong due to there is not node label for kube_pod_status_ready

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Normal Normal
    • None
    • 4.12.0
    • None
    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      SDNPodNotReady annotations is "SDN pod {{ $labels.pod }} on node {{ $labels.node }} is not ready.", but there is not "labels.node" for kube_pod_status_read metrics

      # oc -n openshift-sdn get prometheusrules networking-rules -oyaml | grep SDNPodNotReady -C12
      ...
          - alert: SDNPodNotReady
            annotations:
              message: SDN pod {{ $labels.pod }} on node {{ $labels.node }} is not ready.
            expr: |
              kube_pod_status_ready{namespace='openshift-sdn', condition='true'} == 0
            for: 10m
            labels:
              severity: warning
      

      see:

      # token=`oc create token prometheus-k8s -n openshift-monitoring` 
      # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?" --data-urlencode "query=kube_pod_status_ready" | jq
      ...
            {
              "metric": {
                "__name__": "kube_pod_status_ready",
                "condition": "false",
                "container": "kube-rbac-proxy-main",
                "endpoint": "https-main",
                "job": "kube-state-metrics",
                "namespace": "openshift-apiserver",
                "pod": "apiserver-8668766666-6rftj",
                "prometheus": "openshift-monitoring/k8s",
                "service": "kube-state-metrics",
                "uid": "687a5a01-62f6-448b-a706-909fc7bc6872"
              },
              "value": [
                1669025915.468,
                "0"
              ]

       

      https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md

      also shows there is not node label for kube_pod_status_ready

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-11-19-191518

      How reproducible:

      always

      Steps to Reproduce:

      1. see the description
      2.
      3.
      

              apanatto@redhat.com Andrea Panattoni
              juzhao@redhat.com Junqi Zhao
              Zhanqi Zhao Zhanqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: