Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-30274

[2217243] virt-handler memory and cpu usage are hardcoded and set too low for large scale

XMLWordPrintable

    • 5
    • False
    • Hide

      None

      Show
      None
    • ---
    • ---
    • CNV I/U Operators Sprint 242, CNV Virtualization Sprint 244
    • Urgent
    • No

      I'm running a scale regression setup on :
      =========================================
      OpenShift 4.13.2
      OpenShift Virtualization 4.13.1
      OpenShift Container Storage - 4.12.4-rhodf

      this is a large-scale setup with 132 nodes running 6000 RHEL VMs on an external RHCS.
      after powering up 3000 out of 6000 VMs I opened the UI and see thousands of the following warnings regarding virt-handlers memory consumption:

      ============================================================================================================================================
      KubeVirtComponentExceedsRequestedCPU
      25 Jun 2023, 11:58
      Pod virt-handler-hmztx cpu usage exceeds the CPU requested
      View details
      KubeVirtComponentExceedsRequestedCPU
      25 Jun 2023, 11:58
      Pod virt-handler-b7jfm cpu usage exceeds the CPU requested
      View details
      KubeVirtComponentExceedsRequestedMemory
      25 Jun 2023, 11:58
      Container virt-handler in pod virt-handler-66x4l memory usage exceeds the memory requested
      View details

      ============================================================================================================================================

      from oc adm top:

      ============================================================================================================================================

      virt-handler-x46kr 12m 314Mi
      virt-handler-pld9d 21m 315Mi
      virt-handler-sfqnh 17m 316Mi
      virt-handler-dlh4w 26m 317Mi
      virt-handler-lbfj7 17m 317Mi
      virt-handler-tcx9l 24m 319Mi
      virt-handler-fggsg 18m 321Mi
      virt-handler-7gzm8 17m 325Mi
      virt-handler-lk9bp 12m 325Mi
      virt-handler-gcwfh 18m 329Mi
      ============================================================================================================================================

      if you look at the screenshot I attached you will see that because of it the status of OpenShift Virtualization is at a "Degraded" state
      in addition, the thousands of warnings cause the console to significantly slow down and freeze.

      I collected the logs but I found it odd that I could not see the above events via cli, note this is a 44G folder when extracted:
      ============================================================================================================================================

      http://perf148h.perf.lab.eng.bos.redhat.com/share/BZ_logs/virthandler_mem_cpu_too_low.tar.gz

              ibezukh Igor Bezukh
              bbenshab Boaz Ben Shabat
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: