-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
CNV v4.19.9
-
None
-
Quality / Stability / Reliability
-
0.42
-
False
-
-
False
-
None
-
-
Moderate
-
None
Description of problem:
The system enters an state that is hard for the customer to get out of without a VM shutdown or config changes.
Version-Release number of selected component (if applicable):
4.19.9
How reproducible:
Always
Steps to Reproduce:
1. Have some setting that makes the virt-launcher Pod get memory requests, i.e.: customer using higherWorkloadDensity/memoryOvercommitPercentage, but still allows for memory hotplug (don't set requests on the VM itself). 2. Create a ResourceQuota such as below, allowing up to 8Gi apiVersion: v1 kind: ResourceQuota metadata: name: vms namespace: openshift-mtv spec: hard: requests.cpu: "2" requests.memory: 8Gi 2. Create a VM with 1 CPU and 4Gi of RAM (memory.guest = 4Gi) 3. Run the VM 4. Update the VM to 16G (which will request 100% / 150% * 16Gi = 10.6Gi) if overcommit is 150%. 5. Note 10.6Gi > quota of 8Gi 6. Now the controller keeps creating workload updates VMIM forever, and they are all being denied. Canceling migration doesn't work because the controller creates again. kind: VirtualMachineInstanceMigration ... status: conditions: - lastProbeTime: "2025-10-01T23:18:59Z" lastTransitionTime: null status: "True" type: migrationRejectedByResourceQuota 7. Only way out is to shutdown the VM or disable workload updates. This is undesirable for the user.
Actual results:
Keeps trying to migrate and generating tons of errors. Hard to get out of this situation. 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-zfnfw" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-6lv9j" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-5sjrv" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-w8nv7" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-wklbk" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-rvjs9" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-l9pgn" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 7s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-cfdkf" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi 6s Warning FailedCreate virtualmachineinstance/windows-2019 Error creating pod: pods "virt-launcher-windows-2019-l5z46" is forbidden: exceeded quota: vms, requested: requests.memory=17521053377, used: requests.memory=4501933760, limited: requests.memory=8Gi
Expected results:
- Something more user friendly. - Indicate to the user workload update failed with clear reason (in the Console) - Maybe abort once with a proper error and indicate to the user live migration will not be possible due to quota exceeded - Perhaps suggest to shutdown and start, or disable workload update for this VM with clear indication and let user handle it, but stop this loop.