-
Bug
-
Resolution: Done
-
Normal
-
None
-
4.12.0
-
None
-
None
-
Rejected
-
False
-
Description of problem:
SDNPodNotReady annotations is "SDN pod {{ $labels.pod }} on node {{ $labels.node }} is not ready.", but there is not "labels.node" for kube_pod_status_read metrics
# oc -n openshift-sdn get prometheusrules networking-rules -oyaml | grep SDNPodNotReady -C12 ... - alert: SDNPodNotReady annotations: message: SDN pod {{ $labels.pod }} on node {{ $labels.node }} is not ready. expr: | kube_pod_status_ready{namespace='openshift-sdn', condition='true'} == 0 for: 10m labels: severity: warning
see:
# token=`oc create token prometheus-k8s -n openshift-monitoring` # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?" --data-urlencode "query=kube_pod_status_ready" | jq ... { "metric": { "__name__": "kube_pod_status_ready", "condition": "false", "container": "kube-rbac-proxy-main", "endpoint": "https-main", "job": "kube-state-metrics", "namespace": "openshift-apiserver", "pod": "apiserver-8668766666-6rftj", "prometheus": "openshift-monitoring/k8s", "service": "kube-state-metrics", "uid": "687a5a01-62f6-448b-a706-909fc7bc6872" }, "value": [ 1669025915.468, "0" ]
https://github.com/kubernetes/kube-state-metrics/blob/master/docs/pod-metrics.md
also shows there is not node label for kube_pod_status_ready
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-11-19-191518
How reproducible:
always
Steps to Reproduce:
1. see the description 2. 3.