Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-63962

Can't LiveMigrate a VM without cloud-init volume after Storage Migration from HPP to OCS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • CNV v4.19.4
    • CNV v4.19.0
    • Storage Platform
    • None
    • Quality / Stability / Reliability
    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • CNV v4.19.1.rhel9-15
    • CNV Storage 273
    • None

      Description of problem:

      LiveMigration failed with 
      failureReason: 'Live migration failed error encountered during MigrateToURI3 libvirt       api call: virError(Code=8, Domain=10, Message=''invalid argument: NBD URI must       be supplied when migration URI uses UNIX transport method'')'
      
      Was running on BareMetal cluster: bm02-cnvqe-rdu2

      Version-Release number of selected component (if applicable):

      4.19.0

      How reproducible:

      Always, when migrated from HPP to OCS 
      Never, when migrated from OCS to OCS
      
      In both cases, vTPM PVC is on HPP

      Steps to Reproduce:

      1. Create a Win VM with vTPM (In my case, Win 11, with InstanceType and Preference)
      2. Storage migrate the VM 
      3. Run virtctl migrate vm-name

      Actual results:

      $ oc get vmi -A
      NAMESPACE                                            NAME                               AGE   PHASE     IP             NODENAME                                  READY
      storage-migration-test-mtc-storage-class-migration   windows-11-vm-1750258392-6731908   22m   Running   10.131.1.152   cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com   True
      
      
      $ virtctl migrate -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1750258392-6731908 
      VM windows-11-vm-1750258392-6731908 was scheduled to migrate
      
      $ oc get vmim -A -w
      NAMESPACE                                            NAME                             PHASE       VMI
      storage-migration-test-mtc-storage-class-migration   kubevirt-migrate-vm-wchgk        Failed      windows-11-vm-1750258392-6731908
      storage-migration-test-mtc-storage-class-migration   kubevirt-workload-update-9d25r   Succeeded   windows-11-vm-1750258392-6731908
      
      $ oc get vmim -n storage-migration-test-mtc-storage-class-migration kubevirt-migrate-vm-wchgk -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachineInstanceMigration
      metadata:
        annotations:
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/storage-observed-api-version: v1
        creationTimestamp: "2025-06-18T15:15:09Z"
        generateName: kubevirt-migrate-vm-
        generation: 1
        labels:
          kubevirt.io/vmi-name: windows-11-vm-1750258392-6731908
        name: kubevirt-migrate-vm-wchgk
        namespace: storage-migration-test-mtc-storage-class-migration
        resourceVersion: "11140994"
        uid: f516895a-681d-4550-ae4f-3f1e5f913836
      spec:
        vmiName: windows-11-vm-1750258392-6731908
      status:
        migrationState:
          completed: true
          endTimestamp: "2025-06-18T15:15:23Z"
          failed: true
          failureReason: 'Live migration failed error encountered during MigrateToURI3 libvirt
            api call: virError(Code=8, Domain=10, Message=''invalid argument: NBD URI must
            be supplied when migration URI uses UNIX transport method'')'
          migrationConfiguration:
            allowAutoConverge: false
            allowPostCopy: false
            allowWorkloadDisruption: false
            bandwidthPerMigration: "0"
            completionTimeoutPerGiB: 150
            nodeDrainTaintKey: kubevirt.io/drain
            parallelMigrationsPerCluster: 5
            parallelOutboundMigrationsPerNode: 2
            progressTimeout: 150
            unsafeMigrationOverride: false
          migrationUid: f516895a-681d-4550-ae4f-3f1e5f913836
          mode: PreCopy
          sourceNode: cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com
          sourcePersistentStatePVCName: persistent-state-for-windows-11-vm-1750258392-6731908-6xbkc
          sourcePod: virt-launcher-windows-11-vm-1750258392-6731908-l6dhl
          startTimestamp: "2025-06-18T15:15:23Z"
          targetDirectMigrationNodePorts:
            "33571": 0
            "41499": 49152
            "43761": 49153
          targetNode: cnv-qe-12.cnvqe.lab.eng.rdu2.redhat.com
          targetNodeAddress: 10.129.2.58
          targetPersistentStatePVCName: persistent-state-for-windows-11-vm-1750258392-6731908-q9jvp
          targetPod: virt-launcher-windows-11-vm-1750258392-6731908-qc5td
        phase: Failed
        phaseTransitionTimestamps:
        - phase: Pending
          phaseTransitionTimestamp: "2025-06-18T15:15:09Z"
        - phase: Scheduling
          phaseTransitionTimestamp: "2025-06-18T15:15:09Z"
        - phase: Scheduled
          phaseTransitionTimestamp: "2025-06-18T15:15:23Z"
        - phase: PreparingTarget
          phaseTransitionTimestamp: "2025-06-18T15:15:23Z"
        - phase: TargetReady
          phaseTransitionTimestamp: "2025-06-18T15:15:23Z"
        - phase: Failed
          phaseTransitionTimestamp: "2025-06-18T15:15:23Z"
      

      Expected results:

      VM Live Migrates successfully

      Additional info:

      VM:

      $ oc get vm -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1750258392-6731908 -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachine
      metadata:
        annotations:
          kubemacpool.io/transaction-timestamp: "2025-06-18T14:59:21.958726678Z"
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/storage-observed-api-version: v1
        creationTimestamp: "2025-06-18T14:53:12Z"
        finalizers:
        - kubevirt.io/virtualMachineControllerFinalize
        generation: 3
        name: windows-11-vm-1750258392-6731908
        namespace: storage-migration-test-mtc-storage-class-migration
        resourceVersion: "11130059"
        uid: 4716d320-ba5a-4b70-836b-f2820c0df0e1
      spec:
        dataVolumeTemplates:
        - metadata:
            creationTimestamp: null
            name: windows-11-dv-mig-hbbc
            namespace: storage-migration-test-mtc-storage-class-migration
          spec:
            contentType: kubevirt
            source:
              http:
                certConfigMap: artifactory-configmap
                secretRef: cnv-tests-artifactory-secret
                url: <server>/cnv-qe-server-local/cnv-tests/windows-images/win_11_wsl2.qcow2
            storage:
              resources:
                requests:
                  storage: 70Gi
              storageClassName: hostpath-csi-basic
        instancetype:
          kind: VirtualMachineClusterInstancetype
          name: u1.large
        preference:
          kind: VirtualMachineClusterPreference
          name: windows.11
        runStrategy: Always
        template:
          metadata:
            creationTimestamp: null
            labels:
              debugLogs: "true"
              kubevirt.io/domain: windows-11-vm-1750258392-6731908
              kubevirt.io/vm: windows-11-vm-1750258392-6731908
          spec:
            architecture: amd64
            domain:
              devices:
                disks:
                - disk:
                    bus: virtio
                  name: dv-disk
                rng: {}
              machine:
                type: pc-q35-rhel9.6.0
              resources: {}
            evictionStrategy: None
            volumes:
            - dataVolume:
                name: windows-11-dv-mig-hbbc
              name: dv-disk
        updateVolumesStrategy: Migration
      status:
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2025-06-18T15:00:57Z"
          status: "True"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: null
          message: All of the VMI's DVs are bound and ready
          reason: AllDVsReady
          status: "True"
          type: DataVolumesReady
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: LiveMigratable
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: StorageLiveMigratable
        - lastProbeTime: "2025-06-18T14:58:40Z"
          lastTransitionTime: null
          status: "True"
          type: AgentConnected
        created: true
        desiredGeneration: 3
        instancetypeRef:
          controllerRevisionRef:
            name: windows-11-vm-1750258392-6731908-u1.large-v1beta1-c7d86666-df8d-4c21-927c-b7c0bab5ae28-1
          kind: VirtualMachineClusterInstancetype
          name: u1.large
        observedGeneration: 2
        preferenceRef:
          controllerRevisionRef:
            name: windows-11-vm-1750258392-6731908-windows.11-v1beta1-8fbff2a8-b405-42e1-8dc9-b4fbaebb14a9-1
          kind: VirtualMachineClusterPreference
          name: windows.11
        printableStatus: Running
        ready: true
        runStrategy: Always
        volumeSnapshotStatuses:
        - enabled: true
          name: dv-disk
        volumeUpdateState:
          volumeMigrationState:
            migratedVolumes:
            - destinationPVCInfo:
                claimName: windows-11-dv-mig-hbbc
                volumeMode: Block
              sourcePVCInfo:
                claimName: windows-11-dv
                volumeMode: Filesystem
              volumeName: dv-disk
      

      VMI:

      $ oc get vmi -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1750260600-0055554 -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachineInstance
      metadata:
        annotations:
          kubevirt.io/cluster-instancetype-name: u1.large
          kubevirt.io/cluster-preference-name: windows.11
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/nonroot: "true"
          kubevirt.io/storage-observed-api-version: v1
          kubevirt.io/vm-generation: "2"
          vm.kubevirt.io/os: windows
        creationTimestamp: "2025-06-18T15:30:01Z"
        finalizers:
        - kubevirt.io/virtualMachineControllerFinalize
        - foregroundDeleteVirtualMachine
        generation: 37
        labels:
          debugLogs: "true"
          kubevirt.io/domain: windows-11-vm-1750260600-0055554
          kubevirt.io/migrationTargetNodeName: cnv-qe-11.cnvqe.lab.eng.rdu2.redhat.com
          kubevirt.io/nodeName: cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com
          kubevirt.io/vm: windows-11-vm-1750260600-0055554
        name: windows-11-vm-1750260600-0055554
        namespace: storage-migration-test-mtc-storage-class-migration
        ownerReferences:
        - apiVersion: kubevirt.io/v1
          blockOwnerDeletion: true
          controller: true
          kind: VirtualMachine
          name: windows-11-vm-1750260600-0055554
          uid: 222b33d6-4129-49bd-af8c-7f0af9338db0
        resourceVersion: "11160142"
        uid: 1d4d45a1-27f2-4252-b08b-4143f0dd7d47
      spec:
        architecture: amd64
        domain:
          clock:
            timer:
              hpet:
                present: false
              hyperv:
                present: true
              pit:
                present: true
                tickPolicy: delay
              rtc:
                present: true
                tickPolicy: catchup
            utc: {}
          cpu:
            cores: 1
            maxSockets: 8
            model: host-model
            sockets: 2
            threads: 1
          devices:
            autoattachInputDevice: true
            disks:
            - disk:
                bus: virtio
              name: dv-disk
            inputs:
            - bus: usb
              name: default-0
              type: tablet
            interfaces:
            - masquerade: {}
              model: e1000e
              name: default
            rng: {}
            tpm:
              persistent: true
          features:
            acpi:
              enabled: true
            apic:
              enabled: true
            hyperv:
              frequencies:
                enabled: true
              ipi:
                enabled: true
              reenlightenment:
                enabled: true
              relaxed:
                enabled: true
              reset:
                enabled: true
              runtime:
                enabled: true
              spinlocks:
                enabled: true
                spinlocks: 8191
              synic:
                enabled: true
              synictimer:
                direct:
                  enabled: true
                enabled: true
              tlbflush:
                enabled: true
              vapic:
                enabled: true
              vpindex:
                enabled: true
            smm:
              enabled: true
          firmware:
            bootloader:
              efi:
                persistent: true
                secureBoot: true
            uuid: c9744c86-c42a-5afa-87b4-a860a02a0d4b
          machine:
            type: pc-q35-rhel9.6.0
          memory:
            guest: 8Gi
            maxGuest: 32Gi
          resources:
            requests:
              memory: 8Gi
        evictionStrategy: None
        networks:
        - name: default
          pod: {}
        terminationGracePeriodSeconds: 3600
        volumes:
        - dataVolume:
            name: windows-11-dv-mig-b9zg
          name: dv-disk
      status:
        activePods:
          39c20e88-8c2f-4fd5-a0e8-5dab904727f3: cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com
          69126323-af4b-48b8-b493-d92704b2e222: cnv-qe-11.cnvqe.lab.eng.rdu2.redhat.com
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2025-06-18T15:37:22Z"
          status: "True"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: null
          message: All of the VMI's DVs are bound and ready
          reason: AllDVsReady
          status: "True"
          type: DataVolumesReady
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: StorageLiveMigratable
        - lastProbeTime: "2025-06-18T15:35:08Z"
          lastTransitionTime: null
          status: "True"
          type: AgentConnected
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: LiveMigratable
        currentCPUTopology:
          cores: 1
          sockets: 2
          threads: 1
        guestOSInfo:
          id: mswindows
          kernelRelease: "22000"
          kernelVersion: "10.0"
          machine: x86_64
          name: Microsoft Windows
          prettyName: Windows 10 Pro
          version: Microsoft Windows 11
          versionId: "11"
        interfaces:
        - infoSource: domain
          ipAddress: 10.131.1.158
          ipAddresses:
          - 10.131.1.158
          - fd02:0:0:4::ac7
          linkState: up
          mac: 0a:58:0a:81:02:60
          name: default
          podInterfaceName: eth0
          queueCount: 1
        launcherContainerImageVersion: registry.redhat.io/container-native-virtualization/virt-launcher-rhel9@sha256:4055a7a075ac87d73c2b6a836ee593861cef2f3df6a15014bc6897f903523470
        machine:
          type: pc-q35-rhel9.6.0
        memory:
          guestAtBoot: 8Gi
          guestCurrent: 8Gi
          guestRequested: 8Gi
        migrationMethod: BlockMigration
        migrationState:
          completed: true
          endTimestamp: "2025-06-18T15:38:05Z"
          failed: true
          failureReason: 'Live migration failed error encountered during MigrateToURI3 libvirt
            api call: virError(Code=8, Domain=10, Message=''invalid argument: NBD URI must
            be supplied when migration URI uses UNIX transport method'')'
          migrationConfiguration:
            allowAutoConverge: false
            allowPostCopy: false
            allowWorkloadDisruption: false
            bandwidthPerMigration: "0"
            completionTimeoutPerGiB: 150
            nodeDrainTaintKey: kubevirt.io/drain
            parallelMigrationsPerCluster: 5
            parallelOutboundMigrationsPerNode: 2
            progressTimeout: 150
            unsafeMigrationOverride: false
          migrationUid: 74d6b99f-817b-4b8b-be0e-b5664a22db33
          mode: PreCopy
          sourceNode: cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com
          sourcePersistentStatePVCName: persistent-state-for-windows-11-vm-1750260600-0055554-mj22h
          sourcePod: virt-launcher-windows-11-vm-1750260600-0055554-jxz9n
          startTimestamp: "2025-06-18T15:38:05Z"
          targetDirectMigrationNodePorts:
            "36533": 49152
            "41461": 0
            "46195": 49153
          targetNode: cnv-qe-11.cnvqe.lab.eng.rdu2.redhat.com
          targetNodeAddress: 10.128.2.44
          targetPersistentStatePVCName: persistent-state-for-windows-11-vm-1750260600-0055554-25w2z
          targetPod: virt-launcher-windows-11-vm-1750260600-0055554-2jw7h
        migrationTransport: Unix
        nodeName: cnv-qe-10.cnvqe.lab.eng.rdu2.redhat.com
        phase: Running
        phaseTransitionTimestamps:
        - phase: Pending
          phaseTransitionTimestamp: "2025-06-18T15:30:01Z"
        - phase: Scheduling
          phaseTransitionTimestamp: "2025-06-18T15:30:01Z"
        - phase: Scheduled
          phaseTransitionTimestamp: "2025-06-18T15:34:43Z"
        - phase: Running
          phaseTransitionTimestamp: "2025-06-18T15:34:50Z"
        qosClass: Burstable
        runtimeUser: 107
        selinuxContext: system_u:object_r:container_file_t:s0:c196,c417
        topologyHints:
          tscFrequency: 2095077000
        virtualMachineRevisionName: revision-start-vm-222b33d6-4129-49bd-af8c-7f0af9338db0-2
        volumeStatus:
        - name: dv-disk
          persistentVolumeClaimInfo:
            accessModes:
            - ReadWriteMany
            capacity:
              storage: 70Gi
            claimName: windows-11-dv-mig-b9zg
            filesystemOverhead: "0"
            requests:
              storage: "75161927680"
            volumeMode: Block
          target: vda
        - name: persistent-state-for-windows-11-vm-1750260600-0055554-mj22h
          persistentVolumeClaimInfo:
            accessModes:
            - ReadWriteOnce
            claimName: persistent-state-for-windows-11-vm-1750260600-0055554-mj22h
          target: "" 

              akalenyu Alex Kalenyuk
              jpeimer@redhat.com Jenia Peimer
              Jenia Peimer Jenia Peimer
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: