Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-76093

Hit error in VirtualMachineCreation phase after migration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • CNV User Experience
    • Quality / Stability / Reliability
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None

      Description of problem:

      Warm migrate a VM from ESXi 7 host to OCP cluster, failed to create VM after migration, hit error:

      Internal error occurred: failed calling webhook 
      "virtualmachine-admission.ssp.kubevirt.io": failed to call webhook: Post
      "https://virt-template-validator.openshift-cnv.svc:443/virtualmachine-validate?timeout=10s":
       dial tcp 10.131.0.99:8443: i/o timeout

      The VM is starting on target cluster and the VMI is pending status, the virt-launcher pod hasn't been scheduled.

      After half an hour, checked the VM status changed to running on target cluster.

      The VirtualMachineCreation error in migration plan is timeout, this may be caused by the network slowness.

      Version-Release number of selected component (if applicable):

      MTV 2.10.1-2 with CNV 4.18.23

      How reproducible:

      Always/ N%
      

      Steps to Reproduce:

      1) Create a ESXi host provider

      2) Warm migrate VM: mtv-func-suse15sp6 from ESXi host to OCP 4.18 cluster

      3) The plan failed in VirtualMachineCreation phase with error below, more details in attached files: plan-cold-susp15sp6-skip-host-yaml, migration-cold-susp15sp6-skip-host-k48sr-yaml

      4) Check the VM status is Starting, the events as below:

      # oc get vm
      NAME                      AGE     STATUS     READY
      mtv-func-suse15sp6        4m50s   Starting   False
      
      # oc describe vm mtv-func-suse15sp6
      Name:         mtv-func-suse15sp6
      Namespace:    mtv-test
      ...
      Status:
        Conditions:
          Last Probe Time:       2025-11-20T09:09:55Z
          Last Transition Time:  2025-11-20T09:09:55Z
          Message:               virt-launcher pod has not yet been scheduled
          Reason:                PodNotExists
          Status:                False
          Type:                  Ready
          Last Probe Time:       <nil>
          Last Transition Time:  2025-11-20T09:09:55Z
          Message:               failed to create virtual machine pod: Internal error occurred: failed calling webhook "ipam-claims.k8s.cni.cncf.io": failed to call webhook: Post "https://kubevirt-ipam-controller-webhook-service.openshift-cnv.svc:443/mutate-v1-pod?timeout=10s": context deadline exceeded
          Reason:                FailedCreate
          Status:                False
          Type:                  Synchronized
        Created:                 true

      5) Check VMI is Pending, the virt-launcher pod has not been scheduled. The events as below:

      # oc get vmi
      NAME                 AGE     PHASE     IP    NODENAME   READY
      mtv-func-suse15sp6   5m29s   Pending                    False
      
      # oc describe vmi mtv-func-suse15sp6
      Name:         mtv-func-suse15sp6
      Namespace:    mtv-test
      Labels:       app=mtv-func-suse15sp6
      ......
      Status:
        Conditions:
          Last Probe Time:       2025-11-20T09:09:55Z
          Last Transition Time:  2025-11-20T09:09:55Z
          Message:               virt-launcher pod has not yet been scheduled
          Reason:                PodNotExists
          Status:                False
          Type:                  Ready
          Last Probe Time:       <nil>
          Last Transition Time:  2025-11-20T09:09:55Z
          Message:               failed to create virtual machine pod: Internal error occurred: failed calling webhook "ipam-claims.k8s.cni.cncf.io": failed to call webhook: Post "https://kubevirt-ipam-controller-webhook-service.openshift-cnv.svc:443/mutate-v1-pod?timeout=10s": context deadline exceeded
          Reason:                FailedCreate
          Status:                False
          Type:                  Synchronized
      ....
      Events:
        Type     Reason            Age                    From                         Message
        ----     ------            ----                   ----                         -------
        Normal   SuccessfulCreate  5m39s                  disruptionbudget-controller  Created PodDisruptionBudget kubevirt-disruption-budget-n6lbp
        Warning  FailedCreate      3m19s (x4 over 4m49s)  virtualmachine-controller    Error creating pod: Internal error occurred: failed calling webhook "ipam-claims.k8s.cni.cncf.io": failed to call webhook: Post "https://kubevirt-ipam-controller-webhook-service.openshift-cnv.svc:443/mutate-v1-pod?timeout=10s": dial tcp 10.131.0.101:9443: i/o timeout
        Warning  FailedCreate      56s (x12 over 5m29s)   virtualmachine-controller    Error creating pod: Internal error occurred: failed calling webhook "ipam-claims.k8s.cni.cncf.io": failed to call webhook: Post "https://kubevirt-ipam-controller-webhook-service.openshift-cnv.svc:443/mutate-v1-pod?timeout=10s": context deadline exceeded

      7) Check the VM status after half an hour, the VM change to running on target cluster

      # oc get vm
      NAME                      AGE     STATUS    READY
      mtv-func-suse15sp6        54m     Running   True 
      
      # oc get vmi
      NAME                 AGE   PHASE     IP             NODENAME                         READY
      mtv-func-suse15sp6   54m   Running   10.129.2.146   qemtvd-04-9cw8v-worker-0-rwdz4   True

      Actual results:

      In step3, Migration plan failed with error

      Expected results:

      In step3: Check for more times or extend the timeout or update the error message to let the user know the plan is succeeded from mtv level

      Additional info:

       

              yfrimanm Yifat Menchik
              chhu@redhat.com Chenli Hu
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: