-
Bug
-
Resolution: Done-Errata
-
Normal
-
4.14
-
Moderate
-
No
-
2
-
OCPEDGE Sprint 253, OCPEDGE Sprint 254, OCPEDGE Sprint 255, OCPEDGE Sprint 256
-
4
-
False
-
-
-
Done
-
The monitoring system for Single Node OpenShift (SNO) cluster is triggering an alert named "HighOverallControlPlaneCPU" related to excessive control plane CPU utilization. However, this alert is misleading as it assumes a multi-node setup with high availability (HA) considerations, which do not apply to SNO deployment.
The customer is receiving MNO alerts in the SNO cluster. Below are the details:
The vDU with 2xRINLINE card is installed on the SNO node with OCP 4.14.14.
Used hardware: Airframe OE22 2U server CPU Intel(R) Xeon Intel(R) Xeon(R) Gold 6428N SPR-SP S3, (32 cores 64 threads) with 128GB memory.
After all vDU pods became running, a few minutes later the following alert was triggered:
"labels":
,
"annotations": {
"description": "Given three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity.
This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA.
If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load.
To fix this, increase the CPU and memory on your control plane nodes.",
"runbook_url": https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md,
"summary": "CPU utilization across all three control plane nodes is higher than two control plane nodes can sustain;
a single control plane node outage may cause a cascading failure; increase available CPU."
The alert description is misleading since this cluster is SNO, there is no HA in this cluster.
Increasing CPU capacity in SNO cluster is not an option.
Although the CPU usage is high, this alarm is not correct.
MNO and SNO clusters should have separate alert descriptions.
- depends on
-
-
- Closed
-
- relates to
-
-
- Closed
-
- links to
-
RHBA-2024:4479
OpenShift Container Platform 4.14.z bug fix update
[OCPBUGS-31354] [release-4.14] Misleading alert regarding high control plane CPU utilization in Single Node OpenShift (SNO) cluster
4.16 related bug https://issues.redhat.com/browse/OCPBUGS-35831 has been verified. we can continue it on 4.15 and 4.14.
Pre-merge verified, refer to steps of bug https://issues.redhat.com/browse/OCPBUGS-31354?focusedId=24636314&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-24636314,
$ oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.14.0-0.ci.test-2024-07-05-043558-ci-ln-mwbj2yt-latest True False 11m Cluster version is 4.14.0-0.ci.test-2024-07-05-043558-ci-ln-mwbj2yt-latest
$ oc get no
NAME STATUS ROLES AGE VERSION
ip-10-0-58-51.us-west-1.compute.internal Ready control-plane,master,worker 28m v1.27.14+95b99ee
Alerts HighOverallControlPlaneCPU and ExtremelyHighIndividualControlPlaneCPU are fired as expected.
Bulat Zamalutdinov
added a comment - Hi rhn-support-dmoessner
Yes, PR got merged in. I need to backport it from 4.16 to 4.14. I'll try to push people to approve backports asap
Bulat Zamalutdinov
added a comment - Hi rhn-support-dmoessner
Yes, PR got merged in. I need to backport it from 4.16 to 4.14. I'll try to push people to approve backports asap Bulat Zamalutdinov
added a comment - Hey rhn-support-dmoessner
I've closed backport PR because original PR got reverted
You can track the progress at GH LINK. We do have hot debates in it
Bulat Zamalutdinov
added a comment - Hey rhn-support-dmoessner
I've closed backport PR because original PR got reverted
You can track the progress at GH LINK . We do have hot debates in it Bulat Zamalutdinov
added a comment - Hi rhn-support-dmoessner. I'm waiting for last labels to be applied to PR so i can merge it.
Yes, when workload partitioning is enabled we take it into an account to adjust alert threshold. When it's not enabled we provide default values
Bulat Zamalutdinov
added a comment - Hi rhn-support-dmoessner . I'm waiting for last labels to be applied to PR so i can merge it.
Yes, when workload partitioning is enabled we take it into an account to adjust alert threshold. When it's not enabled we provide default values 
Junqi Zhao
added a comment - it is kube-apiserver issue, should be someone from apiserver-qe-team to verify
$ oc -n openshift-kube-apiserver get prometheusrules cpu-utilization -oyaml .. spec: groups: - name: control-plane-cpu-utilization rules: - alert: HighOverallControlPlaneCPU annotations: description: Given three control plane nodes, the overall CPU utilization may only be about 2/3 of all available capacity. This is because if a single control plane node fails, the remaining two must handle the load of the cluster in order to be HA. If the cluster is using more than 2/3 of all capacity, if one control plane node fails, the remaining two are likely to fail when they take the load. To fix this, increase the CPU and memory on your control plane nodes. runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver-operator/ExtremelyHighIndividualControlPlaneCPU.md summary: CPU utilization across all three control plane nodes is higher than two control plane nodes can sustain; a single control plane node outage may cause a cascading failure; increase available CPU. expr: | sum( 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) AND on (instance) label_replace( kube_node_role{role="master"}, "instance", "$1", "node", "(.+)" ) ) / count(kube_node_role{role="master"}) > 60 for: 10m labels: namespace: openshift-kube-apiserver severity: warning
assign to wk2019
Junqi Zhao
added a comment - it is kube-apiserver issue, should be someone from apiserver-qe-team to verify
$ oc -n openshift-kube-apiserver get prometheusrules cpu-utilization -oyaml
..
spec:
groups:
- name: control-plane-cpu-utilization
rules:
- alert: HighOverallControlPlaneCPU
annotations:
description: Given three control plane nodes, the overall CPU utilization
may only be about 2/3 of all available capacity. This is because if a single
control plane node fails, the remaining two must handle the load of the
cluster in order to be HA. If the cluster is using more than 2/3 of all
capacity, if one control plane node fails, the remaining two are likely
to fail when they take the load. To fix this , increase the CPU and memory
on your control plane nodes.
runbook_url: https: //github.com/openshift/runbooks/blob/master/alerts/cluster-kube-apiserver- operator /ExtremelyHighIndividualControlPlaneCPU.md
summary: CPU utilization across all three control plane nodes is higher than
two control plane nodes can sustain; a single control plane node outage
may cause a cascading failure; increase available CPU.
expr: |
sum(
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode= "idle" }[1m])) * 100)
AND on (instance) label_replace( kube_node_role{role= "master" }, "instance" , "$1" , "node" , "(.+)" )
)
/
count(kube_node_role{role= "master" })
> 60
for : 10m
labels:
namespace: openshift-kube-apiserver
severity: warning
assign to wk2019 Bulat Zamalutdinov
added a comment - juzhao@redhat.com
I opened a backport PR with the fix there
It seems working fine, can you verify it as well?
Bulat Zamalutdinov
added a comment - juzhao@redhat.com
I opened a backport PR with the fix there
It seems working fine, can you verify it as well? It seems related to https://issues.redhat.com//browse/OCPBUGS-22117 but I'll let the SNO team triage.
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Important: OpenShift Container Platform 4.14.33 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:4479