Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-70160

Workload update trying to migrate VM with memory hotplug forever if hotplug denied by ResourceQuota

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • CNV v4.19.9
    • CNV Virtualization
    • None
    • Quality / Stability / Reliability
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Moderate
    • None

      Description of problem:

      The system enters an state that is hard for the customer to get out of without a VM shutdown or config changes.

      Version-Release number of selected component (if applicable):

      4.19.9

      How reproducible:

      Always

      Steps to Reproduce:

      1. Have some setting that makes the virt-launcher Pod get memory requests, i.e.: customer using higherWorkloadDensity/memoryOvercommitPercentage, but still allows for memory hotplug (don't set requests on the VM itself).
      
      2. Create a ResourceQuota such as below, allowing up to 8Gi
      
      apiVersion: v1
      kind: ResourceQuota
      metadata:
        name: vms
        namespace: openshift-mtv
      spec:
        hard:
          requests.cpu: "2"
          requests.memory: 8Gi
      
      2. Create a VM with 1 CPU and 4Gi of RAM (memory.guest = 4Gi)
      
      3. Run the VM
      
      4. Update the VM to 16G (which will request 100% / 150% * 16Gi = 10.6Gi) if overcommit is 150%.
      
      5. Note 10.6Gi > quota of 8Gi
      
      6. Now the controller keeps creating workload updates VMIM forever, and they are all being denied. Canceling migration doesn't work because the controller creates again.
      
      kind: VirtualMachineInstanceMigration
      ...
      status:
        conditions:
        - lastProbeTime: "2025-10-01T23:18:59Z"
          lastTransitionTime: null
          status: "True"
          type: migrationRejectedByResourceQuota
      
      
      7. Only way out is to shutdown the VM or disable workload updates. This is undesirable for the user.

      Actual results:

      Keeps trying to migrate and generating tons of errors. Hard to get out of this situation.
      
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-zfnfw" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-6lv9j" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-5sjrv" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-w8nv7" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-wklbk" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-rvjs9" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-l9pgn" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      7s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-cfdkf" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
      6s          Warning   FailedCreate           virtualmachineinstance/windows-2019                              Error creating pod: pods "virt-launcher-windows-2019-l5z46" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi

      Expected results:

      - Something more user friendly. 
      - Indicate to the user workload update failed with clear reason (in the Console)
      - Maybe abort once with a proper error and indicate to the user live migration will not be possible due to quota exceeded
      - Perhaps suggest to shutdown and start, or disable workload update for this VM with clear indication and let user handle it, but stop this loop.

              tnisan@redhat.com Tal Nisan
              rhn-support-gveitmic Germano Veit Michel
              Kedar Bidarkar Kedar Bidarkar
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: