-
Bug
-
Resolution: Done-Errata
-
Critical
-
4.16.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
No
-
None
-
Proposed
-
OCPNODE Sprint 251 (Green)
-
1
-
+
-
In Progress
-
Release Note Not Required
-
None
-
None
-
None
-
None
-
None
Description of problem:
Container process of burstable pods have their cpu affinity not matching with that containers cpuset.cpus.
Version-Release number of selected component (if applicable):
4.16.0
How reproducible:
everytime
Steps to Reproduce:
1. Setup a ocp cluster
2. Label one of the node as worker-cnf node.
3. Create mcp pool for worker-cnf
4. Apply the below Performance profile, where
apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
name: performance
spec:
cpu:
isolated: "0-2"
reserved: "3"
workloadHints:
realTime: true
net:
userLevelNetworking: true
nodeSelector:
node-role.kubernetes.io/worker-cnf: ""
numa:
topologyPolicy: single-numa-node
realTimeKernel:
enabled: false
4. Take any burstable pod (For example ovn-kubenode pod) and select any container in the pod . Under the containers cgroup . Check the cgroup.procs to see the container process pid
5. Execute taskset -pc <pid> to get the container process
6. Check cpuset controller interface file cpuset.cpus
7. Cpu affinity of container process should match with cpuset.cpus
Actual results:
Currently container process's affinity matches with reserved cpus and doesn't have access to all the online cpus>
Example: below is the output of crictl ps to get the container id of ovn-controller container
ce64bd4fbea18 136fb7d44ff690480ae9cc8796c31cfc7c0d9c95c092be11180be478c46b803b 52 minutes ago Running ovn-controller 2 f8802eae98c2d ovnkube-node-kz9tk
The container process pid of the above container is : 2932:
sh-5.1# crictl inspect ce64bd4fbea18 | grep pid
"pid": 2932,
"pids": {
Get the cgroup path of the container:
sh-5.1# crictl inspect ce64bd4fbea18 | grep cgroup
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"cgroupsPath": "kubepods-burstable-pod95dc49f5_b84d_4587_a958_1bc7848fd95b.slice:crio:ce64bd4fbea1873be599b12efa704db236a1bc9626a75123c0d21effff4835ae",
Check the cpuaffinity of the pid 2932
sh-5.1# taskset -pc 2932
pid 2932's current affinity list: 3
Change to container cgroup directory:
cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dc49f5_b84d_4587_a958_1bc7848fd95b.slice/crio-ce64bd4fbea1873be599b12efa704db236a1bc9626a75123c0d21effff4835ae.scope/
sh-5.1# cat cgroup.procs
2932
sh-5.1# cat cpuset.cpus
0-3
Expected results:
Burstable pod's Container Process affinity should not have reserved cpus and it's cpu affinity should match with cpuset.cpus
Additional info:
- is related to
-
OCPBUGS-27834 Burstable pods have reserved cpu affinity when performance profile is applied
-
- Closed
-
- links to
-
RHEA-2024:0041
OpenShift Container Platform 4.16.z bug fix update