Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-27834

Burstable pods have reserved cpu affinity when performance profile is applied

XMLWordPrintable

    • Critical
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • 5/3: QE are testing with latest RHEL 9.4 to see if the issue is resolved

      Description of problem:

      On 4.16 with Centos stream Cores 4.16(having RHEL9.4), Burstable Pods have reserved cpu affinity when performance profile is applied . By default when performance profile is applied the worker nodes are in cgroupv1 environment . 

      Version-Release number of selected component (if applicable):

          4.16.0-0.nightly-2024-01-23-071601

      How reproducible:

          Everytime

      Steps to Reproduce:

          1. setup a 4.16 cluster with RHCOS9.4
          2. Apply a performance profile as shown below: 
      
      {code:none}
          apiVersion: performance.openshift.io/v2
      kind: PerformanceProfile
      metadata:
        name: performance
      spec:
        cpu:
          isolated: 1,3-11,13,15-23
          reserved: 0,2,12,14
        hugepages:
          defaultHugepagesSize: 1G
          pages:
          - count: 1
            node: 0
            size: 1G
          - count: 128
            node: 1
            size: 2M
        machineConfigPoolSelector:
          machineconfiguration.openshift.io/role: worker-cnf
        nodeSelector:
          node-role.kubernetes.io/worker-cnf: ''
        numa:
          topologyPolicy: single-numa-node
        realTimeKernel:
          enabled: true
        workloadHints:
          highPowerConsumption: true
          perPodPowerManagement: false
          realTime: true
      3. Wait for the nodes to be ready , Check any burstable pod for example ovs pods: 
       
      4. Login to the node and check the cpu affinity of all the burstable pods:

       

      Actual results:

         find /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/  -type f -name "cpuset.cpus" -exec sh -c 'echo $1; cat "$1"' _ {} \;
        /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-ccd741785424a6bcd390710f2af0b72794b9d6fd433490fac33f4bcf135a2583.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-b4a42f6d744ba0f01d2f1021cd4c48019f477eacdd2691518b8d922af830e664.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-30d82ae0b0173dbe4f150d232b50a5f72f7357a776a0b4bd1450c96f13c136ad/cpuset.cpus
      0-23
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/cpuset.cpus
      0-23
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29770f9_bd29_45d3_ac6a_addff93fbb57.slice/crio-b75334c626b0b6fba7de1553e7345a7e0e301aaa960ce0cb9f9177c36e5a152f.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-8f62a6905d5a1889e733866f50d2f224417bae7778b3bb219a915810d8822dd7.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-5423c7659403f45b6cdec01e5c7f4d9c86b58db67b4e7c0aeca5460fbeda09ab.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-e75a28c898b3716d59444166fe9d42a4f3a9b1e36739b2d53dde7cf988015f00.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-e8567fcc6f0addcce6be7fbb4ec19204370e52eb2d79260b58edf65d5b6c14f7.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-4012c6ffd6aab41bd57fbe1fa35f3e0ec4dc6f7d8a754b26a00187491b937244/cpuset.cpus
      0-23
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-65a70d677c7dd50fe805acf206a8145cf12a18f1f8f081ab19c3ef77123c3422.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-ce43dad8d51e99ca6dcc0404df30c44ea878e399c494fb33510ce6a86eb8f8ee.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/crio-15b5b9d5c520061bfe27548117f75e75b0752b0261999ebeb837c99bd98097e8.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c296dd_2ef9_4cb2_99ec_7284f103298b.slice/cpuset.cpus
      0-23
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a21b23_f299_4e5f_9900_ccdbf1eb1bf1.slice/crio-dbaa0baf540c1ccfdd9c0f5f495d4a481110569de4c5e3b27dc6587ae705bf1e.scope/cpuset.cpus
      0,2,12,14
      /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15a21b23_f299_4e5f_9900_ccdbf1eb1bf1.slice/crio-12c49e10d324436150483fb0c7e6eb670d07fb8ecf2983b10c2f1af8ff9e1cc9.scope/cpuset.

      Expected results:

          /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-54134c20334e1598760dc0ea0c4b6002c21d8fe6fa88d008e40a691d76750640.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-92e9d2f3597b5d069e5c9497f1d0454f79f7ccb0420ede964bf48d8d93cc8b24.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/crio-861c0c6b1d46425dfe5ad033f1895c0c3764a9c1a207b1ae442562242610aa8a/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94595293_dddd_419d_90d0_588fdd95c02a.slice/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-e0f5507d81bcb3a99e12f48efa930538ef90b5e3c476fba7566a9e790622daf6/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-1b6666a932315da54a6f5eaf23adbdc9fcb0f1da320c4f8946607d7c942412ce.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-767eaa6b3aeeb1f97f4129c33f3e5b5abe43d4363e02ba1f2e0262946c6471c9.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-2ff939e9a17fc6604053fdba56f737ecc5d60f232df3e570a3b118de472ddbd1.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-d07d70934fd16b7317fe2c71d08b78682197304f4b483e95cbba9c7c312fb6a0.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-de208abde6acc831cd0fc8db31a36b450da8b2009ae1c7242674b4b03fdf32ac.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-e1b430b139a431166cb83861ac386638cda13cdcb4e6727e62cd168d5c93064b.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/crio-ba0f3c8675909f536b938b2d4781f58a3cd7b1722ad21ac78e1867a72717c7b5.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd596fb3_f607_4fb8_b179_818ba7afaf94.slice/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-54bb6e2a0e5b243f5dff0d0db28aef1a82d8d700955f2ccd067c2503f267c91a.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-95eaa9ff025e987c69712f73d2b3c461ba78e1a04d0ecaf74eed8368e8c68298.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f7a6a5dec077a184ccc47965380cfa.slice/crio-4b1d385abcb80f4c30cc1e55a5fc5c9ac6d340db66dc09dcece2d9a2456e728f/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-4843863c7696cf233c8a6c76fdda47190afdcad277eb3628a5f2022505939e71.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-28b6c89ad2131bcdd23cd2d178c3759adef41ecccf52fb26faf08ed2dd2c5b3a.scope/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/crio-7b3a017f667eb1c8becefa7ec388a0f9d7a9a90c94fb5f19b320a65200a8f836/cpuset.cpus
      0-23
       /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f1dbb_75e9_4d51_aad3_00e01073303b.slice/cpuset.cpus
      0-23
      
      

      Additional info:

       1.   Issue occurs only on  CentOS Stream CoreOS release 4.16  and with cgroup version v1 and v2 
       2.   Issue does not occur on RHCOS 4.16 which is having RHEL9.2 OS contents 
       3.   Issue occurs on CentOS Stream CoreOS release 4.16  and with cgroup version v1, v2 and on both runtimes (crun and runc)
       
       

            pehunt@redhat.com Peter Hunt
            mniranja Mallapadi Niranjan
            Mallapadi Niranjan Mallapadi Niranjan
            Martin Sivak
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: