Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14051

kubelet: do not set CPU quota for guaranteed pods

XMLWordPrintable

    • Important
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: A cpu pinned container withing a Guaranteed QoS pod has cgroups quota defined.
      *Consequence*: Rounding and small delays in kernel cpu time accounting might cause throttling of the cpu pinned process even though the quota is set to allow 100% cpu consumption for each allocated cpu.
      *Workaround*: Use a PerformanceProfile and configure the Pod's runtime class and cpu-quota.crio.io: disable annotation to explicitly disable the cpu quota.
      *Result*: The annotation disables the cpu quota and time accounting and removes the source of throttling.
      Show
      *Cause*: A cpu pinned container withing a Guaranteed QoS pod has cgroups quota defined. *Consequence*: Rounding and small delays in kernel cpu time accounting might cause throttling of the cpu pinned process even though the quota is set to allow 100% cpu consumption for each allocated cpu. *Workaround*: Use a PerformanceProfile and configure the Pod's runtime class and cpu-quota.crio.io: disable annotation to explicitly disable the cpu quota. *Result*: The annotation disables the cpu quota and time accounting and removes the source of throttling.
    • Known Issue
    • 2025-02-04: Upstream PR https://github.com/kubernetes/kubernetes/pull/127525 still progressing, high hopes there.

      Description of problem:

      since 4.12 we have the NOHZ_FULL kernel argument added by default and that somehow triggers throttling if quota not disabled

remove quota only in cases, when:

CPU manager runs with the static policy
The pod has guaranteed QoS
The container requested whole CPUs

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      4.12 with apps using Sriov pod has guaranteed QoS

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

      In theory setting CPU CFS quota for CPU pinned pods should not affect performance, but in practice it does cause performance degradation due to the (buggy) way that CFS quota are implemented in the Linux kernel.I think the best fix would be to unset any CPU CFS quota from the pod in the CPU manager when reserved CPUs (CPU sets) are assigned

      Expected results:

      the pod should not have CFS quota defined.

      Additional info:

      My understanding is that if the pod is in guaranteed QoS class and requests integral number of cpus, it should not be subject to CFS quota

              msivak@redhat.com Martin Sivak
              rhn-support-adhingra Anil Dhingra
              Michael Nguyen Michael Nguyen
              Francesco Romani
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

                Created:
                Updated: