Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30329

Excessive node status updates causing high control plane CPU

    XMLWordPrintable

Details

    • No
    • MCO Sprint 250
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      This is a clone of issue OCPBUGS-30285. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-30225. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-29797. The following is the description of the original issue:
      —
      This is a clone of issue OCPBUGS-29713. The following is the description of the original issue:
      —
      Description of problem:

      OCPBUGS-29424 revealed that setting the node status update frequency in kubelet (introduced with OCPBUGS-15583) causes a lot of control plane CPU. 

The reason is the increased frequency of kubelet node status updates will trigger second order effects in all control plane operators that usually trigger on node changes (api server, etcd, PDB guard pod controllers, or any other static pod based machinery).

Reverting the code in OCPBUGS-15583, or manually setting the report/status frequency to 0s causes the CPU to drop immediately.

      Version-Release number of selected component (if applicable):

      any version that OCPBUGS-15583 was backported to, 4.16 down to 4.11 AFAIU

      How reproducible:

      always

      Steps to Reproduce:

      1. create a cluster that contains a fix for OCPBUGS-15583
2. observe the apiserver metrics (eg rate(apiserver_request_total[5m])), those should show abnormal values for pod/configmap GET
    alternatively the rate of node updates is increaed (rate(apiserver_request_total{resource="nodes", subresource="status", verb="PATCH"}[1m])) 

    

      Actual results:

      the node status updates every 10s, which causes high CPU usage on control plane operators and apiserver

      Expected results:

      the node status should not update that frequently, meaning the control plane CPU usage should go down again

      Additional info:

      slack thread with the node team:
https://redhat-internal.slack.com/archives/C02CZNQHGN8/p1708429189987849
    

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-30285 Excessive node status updates causing high control plane CPU

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-30285 Excessive node status updates causing high control plane CPU

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub openshift/machine-config-operator#4246: [release-4.12] OCPBUGS-30329: set nodeStatusReportFrequency

          Web Link RHBA-2024:1265 OpenShift Container Platform 4.12.z bug fix update

          Activity

            People

              team-mco Team MCO
              openshift-crt-jira-prow OpenShift Prow Bot
              Prachiti Talgulkar Prachiti Talgulkar
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: