Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30806

Container process of burstable pods have their cpu affinity not matching with that containers cpuset.cpus after Performance Profile is applied

XMLWordPrintable

      Description of problem:

          Container process of burstable pods have their cpu affinity not matching with that  containers cpuset.cpus.  
      

      Version-Release number of selected component (if applicable):

      4.16.0
          

      How reproducible:

      everytime
          

      Steps to Reproduce:

          1. Setup a ocp cluster 
          2. Label one of the node as worker-cnf node.
          3.  Create mcp pool for worker-cnf 
          4. Apply the below Performance profile, where 
      apiVersion: performance.openshift.io/v2
      kind: PerformanceProfile
      metadata:
        name: performance
      spec:
        cpu:
          isolated: "0-2"
          reserved: "3"
        workloadHints:
          realTime: true
        net:
          userLevelNetworking: true
        nodeSelector:
          node-role.kubernetes.io/worker-cnf: ""
        numa:
          topologyPolicy: single-numa-node
        realTimeKernel:
          enabled: false
      4.  Take any burstable pod (For example ovn-kubenode pod)  and select any container in the pod .  Under the containers cgroup . Check the cgroup.procs to see the container process pid
      5. Execute taskset -pc <pid> to get the container process 
      6. Check cpuset controller interface file cpuset.cpus 
      7. Cpu affinity of container process should match with cpuset.cpus 
      
      
      Actual results:
      Currently container process's affinity matches with reserved cpus and doesn't have access to all the online cpus> 
      
      Example: below is the output of crictl ps to get the container id of ovn-controller container
      ce64bd4fbea18       136fb7d44ff690480ae9cc8796c31cfc7c0d9c95c092be11180be478c46b803b                                                         52 minutes ago      Running             ovn-controller                       2                   f8802eae98c2d       ovnkube-node-kz9tk
      
      The container process pid of the above container is : 2932:
      
      sh-5.1# crictl inspect ce64bd4fbea18 | grep pid
          "pid": 2932,
                "pids": {
      
      Get the cgroup path of the container:
      
      sh-5.1# crictl inspect ce64bd4fbea18 | grep cgroup
                "destination": "/sys/fs/cgroup",
                "type": "cgroup",
                "source": "cgroup",
              "cgroupsPath": "kubepods-burstable-pod95dc49f5_b84d_4587_a958_1bc7848fd95b.slice:crio:ce64bd4fbea1873be599b12efa704db236a1bc9626a75123c0d21effff4835ae",
      
      Check the cpuaffinity of the pid 2932
      sh-5.1# taskset -pc 2932
      pid 2932's current affinity list: 3
      
      Change to container cgroup directory:
      cd /sys/fs/cgroup/cpuset/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95dc49f5_b84d_4587_a958_1bc7848fd95b.slice/crio-ce64bd4fbea1873be599b12efa704db236a1bc9626a75123c0d21effff4835ae.scope/
      sh-5.1# cat cgroup.procs
      2932
      sh-5.1# cat cpuset.cpus
      0-3
      
      
      

       

      Expected results:

      Burstable pod's Container Process affinity should  not have reserved cpus and it's cpu affinity should match with cpuset.cpus 
      
          

      Additional info:

          

            pehunt@redhat.com Peter Hunt
            mniranja Mallapadi Niranjan
            Sunil Choudhary Sunil Choudhary
            Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: