-
Bug
-
Resolution: Done
-
Major
-
None
-
4.16.0
-
Critical
-
No
-
Rejected
-
False
-
-
Release Note Not Required
-
In Progress
-
-
5/3: QE are testing with latest RHEL 9.4 to see if the issue is resolved
-
Description of problem:
On 4.16 with Centos stream Cores 4.16(having RHEL9.4), Burstable Pods have reserved cpu affinity when performance profile is applied . By default when performance profile is applied the worker nodes are in cgroupv1 environment .
Version-Release number of selected component (if applicable):
4.16.0-0.nightly-2024-01-23-071601
How reproducible:
Everytime
Steps to Reproduce:
1. setup a 4.16 cluster with RHCOS9.4 2. Apply a performance profile as shown below: {code:none} apiVersion: performance.openshift.io/v2 kind: PerformanceProfile metadata: name: performance spec: cpu: isolated: 1,3-11,13,15-23 reserved: 0,2,12,14 hugepages: defaultHugepagesSize: 1G pages: - count: 1 node: 0 size: 1G - count: 128 node: 1 size: 2M machineConfigPoolSelector: machineconfiguration.openshift.io/role: worker-cnf nodeSelector: node-role.kubernetes.io/worker-cnf: '' numa: topologyPolicy: single-numa-node realTimeKernel: enabled: true workloadHints: highPowerConsumption: true perPodPowerManagement: false realTime: true
3. Wait for the nodes to be ready , Check any burstable pod for example ovs pods: 4. Login to the node and check the cpu affinity of all the burstable pods:
Actual results:
find /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/ -type f -name "cpuset.cpus" -exec sh -c 'echo $1; cat "$1"' _ {} \;
/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-ccd741785424a6bcd390710f2af0b72794b9d6fd433490fac33f4bcf135a2583.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-b4a42f6d744ba0f01d2f1021cd4c48019f477eacdd2691518b8d922af830e664.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-30d82ae0b0173dbe4f150d232b50a5f72f7357a776a0b4bd1450c96f13c136ad/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-b75334c626b0b6fba7de1553e7345a7e0e301aaa960ce0cb9f9177c36e5a152f.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-8f62a6905d5a1889e733866f50d2f224417bae7778b3bb219a915810d8822dd7.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-5423c7659403f45b6cdec01e5c7f4d9c86b58db67b4e7c0aeca5460fbeda09ab.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-e75a28c898b3716d59444166fe9d42a4f3a9b1e36739b2d53dde7cf988015f00.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-e8567fcc6f0addcce6be7fbb4ec19204370e52eb2d79260b58edf65d5b6c14f7.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-4012c6ffd6aab41bd57fbe1fa35f3e0ec4dc6f7d8a754b26a00187491b937244/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-65a70d677c7dd50fe805acf206a8145cf12a18f1f8f081ab19c3ef77123c3422.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-ce43dad8d51e99ca6dcc0404df30c44ea878e399c494fb33510ce6a86eb8f8ee.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-15b5b9d5c520061bfe27548117f75e75b0752b0261999ebeb837c99bd98097e8.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a21b23_f299_4e5f_9900_ccdbf1eb1bf1.slice/crio-dbaa0baf540c1ccfdd9c0f5f495d4a481110569de4c5e3b27dc6587ae705bf1e.scope/cpuset.cpus 0,2,12,14 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a21b23_f299_4e5f_9900_ccdbf1eb1bf1.slice/crio-12c49e10d324436150483fb0c7e6eb670d07fb8ecf2983b10c2f1af8ff9e1cc9.scope/cpuset.
Expected results:
/sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-54134c20334e1598760dc0ea0c4b6002c21d8fe6fa88d008e40a691d76750640.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-92e9d2f3597b5d069e5c9497f1d0454f79f7ccb0420ede964bf48d8d93cc8b24.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-861c0c6b1d46425dfe5ad033f1895c0c3764a9c1a207b1ae442562242610aa8a/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-e0f5507d81bcb3a99e12f48efa930538ef90b5e3c476fba7566a9e790622daf6/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-1b6666a932315da54a6f5eaf23adbdc9fcb0f1da320c4f8946607d7c942412ce.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-767eaa6b3aeeb1f97f4129c33f3e5b5abe43d4363e02ba1f2e0262946c6471c9.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-2ff939e9a17fc6604053fdba56f737ecc5d60f232df3e570a3b118de472ddbd1.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-d07d70934fd16b7317fe2c71d08b78682197304f4b483e95cbba9c7c312fb6a0.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-de208abde6acc831cd0fc8db31a36b450da8b2009ae1c7242674b4b03fdf32ac.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-e1b430b139a431166cb83861ac386638cda13cdcb4e6727e62cd168d5c93064b.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-ba0f3c8675909f536b938b2d4781f58a3cd7b1722ad21ac78e1867a72717c7b5.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-54bb6e2a0e5b243f5dff0d0db28aef1a82d8d700955f2ccd067c2503f267c91a.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-95eaa9ff025e987c69712f73d2b3c461ba78e1a04d0ecaf74eed8368e8c68298.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-4b1d385abcb80f4c30cc1e55a5fc5c9ac6d340db66dc09dcece2d9a2456e728f/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-4843863c7696cf233c8a6c76fdda47190afdcad277eb3628a5f2022505939e71.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-28b6c89ad2131bcdd23cd2d178c3759adef41ecccf52fb26faf08ed2dd2c5b3a.scope/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-7b3a017f667eb1c8becefa7ec388a0f9d7a9a90c94fb5f19b320a65200a8f836/cpuset.cpus 0-23 /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/cpuset.cpus 0-23
Additional info:
1. Issue occurs only on CentOS Stream CoreOS release 4.16 and with cgroup version v1 and v2 2. Issue does not occur on RHCOS 4.16 which is having RHEL9.2 OS contents 3. Issue occurs on CentOS Stream CoreOS release 4.16 and with cgroup version v1, v2 and on both runtimes (crun and runc)
- is related to
-
OCPBUGS-15102 All burstable pods run with the reserved cpu affinity mask when PerformanceProfile is applied
- Closed
- relates to
-
OCPBUGS-30806 Container process of burstable pods have their cpu affinity not matching with that containers cpuset.cpus after Performance Profile is applied
- Closed
-
OCPBUGS-28732 An invalid isolated partition cgroups error when cpu load-balance disabled
- Closed