-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
Quality / Stability / Reliability
-
0.42
-
False
-
-
False
-
None
-
-
None
This is a backport request for CNV-68789 to 4.18 (incl all intermediate versions).
Description of problem:
live migrating more then 45 VM from 2 nodes(each with 22-27 vms) generates below error
Error migrating VirtualMachine Internal error occurred: Internal error occurred: failed calling webhook "migration-create-validator.kubevirt.io": failed to call webhook: Post "https://virt-api.openshift-cnv.svc:443/migration-validate-create?timeout=10s": context deadline exceeded
Version-Release number of selected component (if applicable):
openshift-cnv kubevirt-hyperconverged-operator.v4.18.8 OpenShift Virtualization 4.18.8 kubevirt-hyperconverged-operator.v4.18.3 Succeeded openshift-ovirt-infra node-healthcheck-operator.v0.9.0 Node Health Check Operator 0.9.0 node-healthcheck-operator.v0.8.2 Succeeded
How reproducible:
virtctl migrate "$vm_name" -n "$NAMESPACE"and pass in list of 45 vm names from 2 nodes
Steps to Reproduce:
1.migrate 45 or more vms from 2 nodes
2.for ((i=0; i<COUNT && i<${#all_vms[@]}; i++)); do
vm="${all_vms[$i]}"
migrate_vm "$vm" & # virtctl migrate "$vm_name" -n "$NAMESPACE" done
to migrate vm in parallel
3.Got below error Error migrating VirtualMachine Internal error occurred: Internal error occurred: failed calling webhook "migration-create-validator.kubevirt.io": failed to call webhook: Post "https://virt-api.openshift-cnv.svc:443/migration-validate-create?timeout=10s": context deadline exceeded
Actual results:
Error migrating VirtualMachine Internal error occurred: Internal error occurred: failed calling webhook "migration-create-validator.kubevirt.io": failed to call webhook: Post "https://virt-api.openshift-cnv.svc:443/migration-validate-create?timeout=10s": context deadline exceeded
Expected results:
no error
Additional info:
Resource usage in the cluster [root@d41-h18-000-r660 yog]# oc adm top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% d41-h19-000-r660 925m 0% 22458Mi 4% d41-h20-000-r660 2617m 2% 24232Mi 4% e09-h01-000-r660 3763m 3% 39543Mi 7% e09-h02-000-r660 11681m 10% 50660Mi 9% e09-h03-000-r660 12460m 11% 54687Mi 10% e09-h04-000-r660 15008m 13% 47911Mi 9% e09-h05-000-r660 18194m 16% 59571Mi 11% e09-h06-000-r660 24572m 22% 70878Mi 13% e09-h07-000-r660 1376m 1% 8936Mi 1% e09-h08-000-r660 23802m 21% 67765Mi 13% e09-h09-000-r660 23483m 21% 68279Mi 13% e09-h10-000-r660 26363m 23% 71153Mi 13% e09-h12-000-r660 22659m 20% 69852Mi 13% e10-h23-000-r660 25293m 22% 78639Mi 15% e10-h24-000-r660 25406m 22% 77632Mi 15% e10-h26-000-r660 25406m 22% 68436Mi 13% e10-h27-000-r660 24041m 21% 69175Mi 13% e10-h29-000-r660 24343m 21% 64958Mi 12% Before migration,VMs are distributed on 13 nodes(out of 15). vm-count node-name 27 e09-h03-000-r660 26 e09-h05-000-r660 26 e09-h06-000-r660 27 e09-h07-000-r660 26 e09-h08-000-r660 25 e09-h09-000-r660 27 e09-h10-000-r660 26 e09-h12-000-r660 27 e10-h23-000-r660 26 e10-h24-000-r660 26 e10-h26-000-r660 26 e10-h27-000-r660 25 e10-h29-000-r660
