-
Bug
-
Resolution: Duplicate
-
Undefined
-
None
-
4.14.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
Important
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Performing the hardware reboot on each MNO nodes the tuned enters the following state:
NAMESPACE NAME TUNED APPLIED DEGRADED AGE
openshift-cluster-node-tuning-operator master0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 10h
openshift-cluster-node-tuning-operator master1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 12h
openshift-cluster-node-tuning-operator master2.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 11h
openshift-cluster-node-tuning-operator worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True True 11h
openshift-cluster-node-tuning-operator worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True False 11h
Version-Release number of selected component (if applicable):
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.14.14 True False 11h Cluster version is 4.14.14
How reproducible:
80%
Steps to Reproduce:
1.Apply the PAO and Tuned patch to the cluster highlited in the following support case: https://access.redhat.com/support/cases/#/case/03708995/discussion?commentId=a0a6R00000WCpsBQAT
2. Perform reboot of each individual node in the cluster for 24h
3.
Actual results:
NAMESPACE NAME TUNED APPLIED DEGRADED AGE
openshift-cluster-node-tuning-operator master0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 10h
openshift-cluster-node-tuning-operator master1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 12h
openshift-cluster-node-tuning-operator master2.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 11h
openshift-cluster-node-tuning-operator worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True True 11h
openshift-cluster-node-tuning-operator worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True False 11h
apiVersion: tuned.openshift.io/v1
kind: Profile
metadata:
creationTimestamp: "2024-03-19T19:20:50Z"
generation: 4
name: worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net
namespace: openshift-cluster-node-tuning-operator
ownerReferences:
- apiVersion: tuned.openshift.io/v1
blockOwnerDeletion: true
controller: true
kind: Tuned
name: default
uid: 5ab6d677-ef3e-4eaf-98c5-8aa3038c9f5f
resourceVersion: "138405"
uid: dedf4e04-a88c-4961-98a6-0b971653f5f2
spec:
config:
debug: false
providerName: baremetalhost
tunedConfig: {}
tunedProfile: rdpmc-patch-worker
status:
conditions:
- lastTransitionTime: "2024-03-19T19:58:08Z"
message: TuneD profile applied.
reason: AsExpected
status: "True"
type: Applied
- lastTransitionTime: "2024-03-19T21:34:23Z"
message: 'TuneD daemon issued one or more error message(s) during profile application.
TuneD stderr: ERROR tuned.plugins.plugin_scheduler: Failed to set affinity
of PID 1110 to ''[29]'': [Errno 22] Invalid argument'
reason: TunedError
status: "True"
type: Degraded
tunedProfile: rdpmc-patch-worker $ omc logs -n openshift-cluster-node-tuning-operator tuned-4p4qm | grep ERROR
2024-03-19T21:34:20.242693269Z 2024-03-19 21:34:20,242 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 476 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/476/smp_affinity'
2024-03-19T21:34:20.242855820Z 2024-03-19 21:34:20,242 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 493 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/493/smp_affinity'
2024-03-19T21:34:20.243051936Z 2024-03-19 21:34:20,243 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 510 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/510/smp_affinity'
2024-03-19T21:34:20.243449807Z 2024-03-19 21:34:20,243 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 517 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/517/smp_affinity'
2024-03-19T21:34:20.243654335Z 2024-03-19 21:34:20,243 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 519 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/519/smp_affinity'
2024-03-19T21:34:20.244274428Z 2024-03-19 21:34:20,243 ERROR tuned.plugins.plugin_scheduler: Failed to set SMP affinity of IRQ 595 to 'ffffffff,ffffffff': [Errno 2] No such file or directory: '/proc/irq/595/smp_affinity'
2024-03-19T21:34:20.403920714Z 2024-03-19 21:34:20,403 ERROR tuned.plugins.plugin_scheduler: Failed to set affinity of PID 1110 to '[29]': [Errno 22] Invalid argument
2024-03-19T21:34:25.816277984Z 2024-03-19 21:34:25,816 ERROR tuned.plugins.plugin_scheduler: Failed to set affinity of PID 1110 to '[29]': [Errno 22] Invalid argument
Expected results:
NAMESPACE NAME TUNED APPLIED DEGRADED AGE openshift-cluster-node-tuning-operator master0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 11h openshift-cluster-node-tuning-operator master1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 13h openshift-cluster-node-tuning-operator master2.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-master True False 13h openshift-cluster-node-tuning-operator worker0.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True False 13h openshift-cluster-node-tuning-operator worker1.winterfell-mno-2.lab.neat.nsn-rdnet.net rdpmc-patch-worker True False 13h
Additional info:
- duplicates
-
OCPBUGS-31844 tuned: tuned breaks dynamic IRQ affinity
-
- Closed
-
- is incorporated by
-
OCPBUGS-26400 tuned: tuned breaks dynamic IRQ affinity
-
- Closed
-