Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-32022

ovnkube-controller high cpu usage during eip resources creation at scale

XMLWordPrintable

    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      During EIP scale testing, for creating 24000 EIPs on a 120 node baremetal OCP environment, we observed ovnkube-controller CPU usage linearly increased from 4% to 800%. Avergae CPU usage during the test was 360% and max was 1060%. 
      
      Test creates 24000 namespaces wherein for each namespace it creates 1 EIP object with 1 EIP address and 1 Deployment with 1 pod replicas. Thus test creates 24000 namespaces, 24000 EIPs, 24000 pods.
      
      Some customers are using 200 EIPs per node. 8 cores CPU usage on the worker during churn can limit the CPU avaialibity for customer workloads
      
      More details about the testing and observations are at https://docs.google.com/document/d/17NGv6pR-3VFVD5hFzpcdYt5hRZChOSIzmm0BPc-Rdx8/edit?usp=sharing 
      
      Environment details -
      OCP deployed on bare metal nodes - 120 workers, 2 infra, 3 masters.
      All nodes have same configuration -
      CPUs: 80   Memory: 384G   NIC bandwidth: 25 gb/s

      Version-Release number of selected component (if applicable):

          4.16

      How reproducible:

      Always    

      Steps to Reproduce:

      Perf & OVN team developed a custom workload for EIP scale testing. We have the OCP deployment now to debug this issue.    

      Actual results:

       

      Expected results:

          

      Additional info:

          

            mkennell@redhat.com Martin Kennelly
            vkommadi@redhat.com VENKATA ANIL kumar KOMMADDI
            Sachin Ninganure Sachin Ninganure
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: