Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14286

ovnkube-network-controller-manager high CPU usage in OVNK

    XMLWordPrintable

Details

    • Yes
    • SDN Sprint 237
    • 1
    • Approved
    • False
    • Hide

      Seems like a scale regression that is unrelated to Interconnect but something we should fix

      Show
      Seems like a scale regression that is unrelated to Interconnect but something we should fix

    Description

      Description of problem:

      Total ovnkube-network-controller-manager CPU usage is 293 cores across the cluster (i.e cluster with 120 worker nodes and each worker node with 8 CPUs), effectively using 2 cores in each node
      
      This is observed while running cluster-density-v2 workload on OVN-IC 120 node environment with Northd Incremental Processing patches i.e
      OVN_IMAGE - quay.io/itssurya/dev-images:ic-scale-ovnk-v2
      CNO image - quay.io/itssurya/dev-images:ic-cno-hack
      I have got high latency with northd single thread CNO image ( quay.io/itssurya/dev-images:ic-cno-hack-1thread ) as well
      
      OVN-IC without incremental processing is using 19 cores across the cluster.
      Slides for this testing https://docs.google.com/presentation/d/16NMDoF52gUb-MybbAtSs16_888nLbshQYCU1Zw3WuvE/edit#slide=id.g24a34730293_0_0
      
      
      Version-Release number of selected component (if applicable):
      OCP 4.14
      OVN_IMAGE - quay.io/itssurya/dev-images:ic-scale-ovnk-v2
      CNO image - quay.io/itssurya/dev-images:ic-cno-hack

      Version-Release number of selected component (if applicable):

      OCP 4.14
      OVN_IMAGE - quay.io/itssurya/dev-images:ic-scale-ovnk-v2
      CNO image - quay.io/itssurya/dev-images:ic-cno-hack

      How reproducible:

       

      Steps to Reproduce:

      1. Deploy 120 node OCP environment on AWS
      2. Apply OVN builds
      3. Run node-denisty (light) and cluster-density-v2 workloads  

      Actual results:

       

      Expected results:

      Total ovnkube-network-controller-manager CPU usage is 19 cores across the cluster on OVN-IC without Incremental Processing

      Additional info:

       

      Attachments

        Activity

          People

            npinaeva@redhat.com Nadia Pinaeva
            vkommadi@redhat.com VENKATA ANIL kumar KOMMADDI
            Anurag Saxena Anurag Saxena
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: