Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17777

[OVN-IC] 34% increase in avg kube-apiserver CPU usage in OVN-IC compared to OpenshiftSDN

    XMLWordPrintable

Details

    • No
    • SDN Sprint 241
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      kube-apiserver cpu usage increased to 34% in OVN IC compared to openshiftSDN. This is observed while running node-density-cni with 80 pods per node configuration on a 120 node environment. Below are the kube-apiserver cpu usage numbers in all the 3 environments -

       

      SDN Avg CPU %   Max CPU %
        master-0 master-1 master-2 Total master-0 master-1 master-2
      OpenshiftSDN 178 95.4 67 340.4 284 151 97.1
      OVN 146 92.5 50.7 289.2 235 129 93.8
      OVN-IC 195 147 114 456 280 230 169

      Avg kube-apiserver CPU usage 34% increase in IC compared to OpenshiftSDN

      Avg kube-apiserver CPU usage 57% increase in IC compared to OVN

      Max kube-apiserver CPU usage is also very high compared to both OpenshiftSDN and OVN.

      kube-apiserver CPU & memory usage grafana panels -

      Openshiftsdn  https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/Tm54HMwv1m7VOD5T739xygzdqZ6sU3sS

      OVN https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/m44tpS4xo2b5NQd4MIctTX0Bx3uDb1kN

      OVN-IC https://grafana.rdu2.scalelab.redhat.com:3000/dashboard/snapshot/G6CHY96rqdwlHOiwAty1El1fNuX5c2Zc

      OCP version 4.14.0-0.nightly-2023-07-30-234232

      I can provide the environment  when the engineer wants to debug.

       

       

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue API-1648 ocp 4.14 kube-apiserver dramatic increase in rate of terminated node watchers

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-18097 OCP 4.14 increased rate of patch nodes requests from node SAs

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link https://github.com/openshift/cluster-network-operator/pull/1971

          Web Link https://github.com/ovn-org/ovn-kubernetes/pull/3807

          Activity

            People

              mkennell@redhat.com Martin Kennelly
              vkommadi@redhat.com VENKATA ANIL kumar KOMMADDI
              Anurag Saxena Anurag Saxena
              Nadia Pinaeva, Tim Rozet
              Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: