Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14051

kubelet: do not set CPU quota for guaranteed pods

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • Hide
      2025-04-22: Fix will be included only in v1.33 / OCP 4.20 . 4.19 should be documented as a known issue
      2025-02-04: Upstream PR https://github.com/kubernetes/kubernetes/pull/127525 still progressing, high hopes there.
      Show
      2025-04-22: Fix will be included only in v1.33 / OCP 4.20 . 4.19 should be documented as a known issue 2025-02-04: Upstream PR https://github.com/kubernetes/kubernetes/pull/127525 still progressing, high hopes there.
    • None
    • None
    • None
    • Done
    • Known Issue
    • Hide
      Previously, if a CPU-pinned container within a Guaranteed QoS pod has cgroups quota defined, rounding and small delays in kernel CPU time accounting could cause throttling of the CPU-pinned process, even if the the quota is set to allow 100% consumption for each allocated CPU. With this release, when `cpu-manager-policy=static` and the qualifications for static CPU assignment are satisfied, that is containers have Guaranteed QOS with integer CPU requests, CFS quota is disabled.

      If you have pods in earlier {product-title} versions that meet these criteria, you can prevent the throttling by using a performance profile and configuring the pod runtime class and `cpu-quota.crio.io: disable` annotation to explicitly disable the CPU quota. The annotation disables the CPU quota and time accounting and removes the source of throttling. For more information see xref:../../scalability_and_performance/cnf-provisioning-low-latency-workloads#cnf-node-tuning-operator-disabling-cpu-load-balancing-for-dpdk_cnf-provisioning-low-latency[Disabling CPU CFS quota]. (link:https://issues.redhat.com/browse/OCPBUGS-14051[OCPBUGS-14051])
      Show
      Previously, if a CPU-pinned container within a Guaranteed QoS pod has cgroups quota defined, rounding and small delays in kernel CPU time accounting could cause throttling of the CPU-pinned process, even if the the quota is set to allow 100% consumption for each allocated CPU. With this release, when `cpu-manager-policy=static` and the qualifications for static CPU assignment are satisfied, that is containers have Guaranteed QOS with integer CPU requests, CFS quota is disabled. If you have pods in earlier {product-title} versions that meet these criteria, you can prevent the throttling by using a performance profile and configuring the pod runtime class and `cpu-quota.crio.io: disable` annotation to explicitly disable the CPU quota. The annotation disables the CPU quota and time accounting and removes the source of throttling. For more information see xref:../../scalability_and_performance/cnf-provisioning-low-latency-workloads#cnf-node-tuning-operator-disabling-cpu-load-balancing-for-dpdk_cnf-provisioning-low-latency[Disabling CPU CFS quota]. (link: https://issues.redhat.com/browse/OCPBUGS-14051 [ OCPBUGS-14051 ])
    • None
    • None
    • None
    • None

      Description of problem:

      since 4.12 we have the NOHZ_FULL kernel argument added by default and that somehow triggers throttling if quota not disabled

remove quota only in cases, when:

CPU manager runs with the static policy
The pod has guaranteed QoS
The container requested whole CPUs

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      4.12 with apps using Sriov pod has guaranteed QoS

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

      In theory setting CPU CFS quota for CPU pinned pods should not affect performance, but in practice it does cause performance degradation due to the (buggy) way that CFS quota are implemented in the Linux kernel.I think the best fix would be to unset any CPU CFS quota from the pod in the CPU manager when reserved CPUs (CPU sets) are assigned

      Expected results:

      the pod should not have CFS quota defined.

      Additional info:

      My understanding is that if the pod is in guaranteed QoS class and requests integral number of cpus, it should not be subject to CFS quota

              msivak@redhat.com Martin Sivak
              rhn-support-adhingra Anil Dhingra
              None
              Francesco Romani
              Mallapadi Niranjan Mallapadi Niranjan
              None
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated: