Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-69346

[4.19] VMIM is Pending sometimes (for Windows VM with vTPM that was previously Storage Migrated)

XMLWordPrintable

    • Quality / Stability / Reliability
    • 5
    • False
    • Hide

      None

      Show
      None
    • False
    • CNV v4.19.9.rhel9-1
    • CNV Storage 276, CNV Storage 277
    • None

      Description of problem:

      Windows 11 VM with vTPM was sucessfully Storage Live Migrated 
      from hostpath-csi-basic to ocs-storagecluster-ceph-rbd-virtualization storage class.
      
      When we try to Compute Live Migrate this VM, 
      VMIM gets in Pending state right after creation, with no explanation.
      
      Target virt-launcher pod is not getting created.

      Version-Release number of selected component (if applicable):

      CNV v4.20.0, 4.19.0

      How reproducible:

      Could reproduce 2 out of 10 times, but then no repro in the next 15 times

      Steps to Reproduce:

      1. Create a Win11 VM with vtpm
      2. Storage Live Migrate with MTC
      3. Try to compute Live Migrate 
      

      Actual results:

      Sometimes, VMIM is Pending with no info of why is that happening

      Expected results:

      VMIM always succeeds

      Additional info:

      Attached CNV must-gather that was collected on failure of the automated Tier3 test.

      VMIM:

      $ oc get vmim -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1757243511-8243139 -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachineInstanceMigration
      metadata:
        annotations:
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/storage-observed-api-version: v1
        creationTimestamp: "2025-09-07T11:42:33Z"
        finalizers:
        - kubevirt.io/migrationJobFinalize
        generation: 1
        labels:
          kubevirt.io/vmi-name: windows-11-vm-1757243511-8243139
        name: windows-11-vm-1757243511-8243139
        namespace: storage-migration-test-mtc-storage-class-migration
        resourceVersion: "22432582"
        uid: 7faf5f80-ae53-41f2-8ec8-125551c22379
      spec:
        vmiName: windows-11-vm-1757243511-8243139
      status:
        migrationState:
          sourcePod: virt-launcher-windows-11-vm-1757243511-8243139-zgsxb
        phase: Pending
        phaseTransitionTimestamps:
        - phase: Pending
          phaseTransitionTimestamp: "2025-09-07T11:42:33Z"
      
      $ oc get pods -n storage-migration-test-mtc-storage-class-migration
      NAME                                                   READY   STATUS    RESTARTS   AGE
      virt-launcher-windows-11-vm-1757243511-8243139-zgsxb   2/2     Running   0          2m59s
      

      VM:

      $ oc get vm -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1757243511-8243139 -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachine
      metadata:
        annotations:
          kubemacpool.io/transaction-timestamp: "2025-09-07T11:40:34.864331864Z"
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/storage-observed-api-version: v1
        creationTimestamp: "2025-09-07T11:11:53Z"
        finalizers:
        - kubevirt.io/virtualMachineControllerFinalize
        generation: 3
        name: windows-11-vm-1757243511-8243139
        namespace: storage-migration-test-mtc-storage-class-migration
        resourceVersion: "22499461"
        uid: 77da8433-9d37-47fa-95ea-064d55ad49f8
      spec:
        dataVolumeTemplates:
        - metadata:
            creationTimestamp: null
            name: windows-11-dv-mig-l8hs
            namespace: storage-migration-test-mtc-storage-class-migration
          spec:
            source:
              http:
                certConfigMap: artifactory-configmap
                secretRef: cnv-tests-artifactory-secret
                url: https://cnv-qe-artifactory.apps.int.prod-stable-spoke1-dc-iad2.itup.redhat.com/artifactory/cnv-qe-server-local/cnv-tests/windows-images/win_11_wsl2.qcow2
            storage:
              resources:
                requests:
                  storage: 70Gi
              storageClassName: ocs-storagecluster-ceph-rbd-virtualization
        instancetype:
          kind: VirtualMachineClusterInstancetype
          name: u1.large
        preference:
          kind: VirtualMachineClusterPreference
          name: windows.11
        runStrategy: Always
        template:
          metadata:
            creationTimestamp: null
            labels:
              debugLogs: "true"
              kubevirt.io/domain: windows-11-vm-1757243511-8243139
              kubevirt.io/vm: windows-11-vm-1757243511-8243139
          spec:
            architecture: amd64
            domain:
              devices:
                disks:
                - disk:
                    bus: virtio
                  name: dv-disk
                rng: {}
              firmware:
                uuid: 4d20f349-74ac-4035-99e9-c1be74e5c3a1
              machine:
                type: pc-q35-rhel9.6.0
              resources: {}
            evictionStrategy: None
            volumes:
            - dataVolume:
                name: windows-11-dv-mig-l8hs
              name: dv-disk
        updateVolumesStrategy: Migration
      status:
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2025-09-07T13:09:23Z"
          status: "True"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: null
          message: All of the VMI's DVs are bound and ready
          reason: AllDVsReady
          status: "True"
          type: DataVolumesReady
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: LiveMigratable
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: StorageLiveMigratable
        - lastProbeTime: "2025-09-07T11:39:42Z"
          lastTransitionTime: null
          status: "True"
          type: AgentConnected
        created: true
        desiredGeneration: 3
        instancetypeRef:
          controllerRevisionRef:
            name: windows-11-vm-1757243511-8243139-u1.large-v1beta1-f4b2794e-66bc-4127-b98d-affcee971400-1
          kind: VirtualMachineClusterInstancetype
          name: u1.large
        observedGeneration: 2
        preferenceRef:
          controllerRevisionRef:
            name: windows-11-vm-1757243511-8243139-windows.11-v1beta1-8d4bebe2-f9c9-464a-80a5-de7d5da6fabc-1
          kind: VirtualMachineClusterPreference
          name: windows.11
        printableStatus: Running
        ready: true
        runStrategy: Always
        volumeSnapshotStatuses:
        - enabled: true
          name: dv-disk
        volumeUpdateState:
          volumeMigrationState:
            migratedVolumes:
            - destinationPVCInfo:
                claimName: windows-11-dv-mig-l8hs
                volumeMode: Block
              sourcePVCInfo:
                claimName: windows-11-dv
                volumeMode: Filesystem
              volumeName: dv-disk
      

      VMI:

      $ oc get vmi -n storage-migration-test-mtc-storage-class-migration windows-11-vm-1757243511-8243139 -oyaml
      apiVersion: kubevirt.io/v1
      kind: VirtualMachineInstance
      metadata:
        annotations:
          kubevirt.io/cluster-instancetype-name: u1.large
          kubevirt.io/cluster-preference-name: windows.11
          kubevirt.io/latest-observed-api-version: v1
          kubevirt.io/nonroot: "true"
          kubevirt.io/storage-observed-api-version: v1
          kubevirt.io/vm-generation: "2"
          vm.kubevirt.io/os: windows
        creationTimestamp: "2025-09-07T11:11:54Z"
        finalizers:
        - kubevirt.io/virtualMachineControllerFinalize
        - foregroundDeleteVirtualMachine
        generation: 310
        labels:
          debugLogs: "true"
          kubevirt.io/domain: windows-11-vm-1757243511-8243139
          kubevirt.io/nodeName: cnvqe-075.lab.eng.tlv2.redhat.com
          kubevirt.io/vm: windows-11-vm-1757243511-8243139
        name: windows-11-vm-1757243511-8243139
        namespace: storage-migration-test-mtc-storage-class-migration
        ownerReferences:
        - apiVersion: kubevirt.io/v1
          blockOwnerDeletion: true
          controller: true
          kind: VirtualMachine
          name: windows-11-vm-1757243511-8243139
          uid: 77da8433-9d37-47fa-95ea-064d55ad49f8
        resourceVersion: "22499468"
        uid: db9602b3-8242-43a4-bb8c-3a065ac10ece
      spec:
        architecture: amd64
        domain:
          clock:
            timer:
              hpet:
                present: false
              hyperv:
                present: true
              pit:
                present: true
                tickPolicy: delay
              rtc:
                present: true
                tickPolicy: catchup
            utc: {}
          cpu:
            cores: 1
            maxSockets: 8
            model: host-model
            sockets: 2
            threads: 1
          devices:
            autoattachInputDevice: true
            disks:
            - disk:
                bus: virtio
              name: dv-disk
            inputs:
            - bus: usb
              name: default-0
              type: tablet
            interfaces:
            - masquerade: {}
              model: e1000e
              name: default
            rng: {}
            tpm:
              persistent: true
          features:
            acpi:
              enabled: true
            apic:
              enabled: true
            hyperv:
              frequencies:
                enabled: true
              ipi:
                enabled: true
              reenlightenment:
                enabled: true
              relaxed:
                enabled: true
              reset:
                enabled: true
              runtime:
                enabled: true
              spinlocks:
                enabled: true
                spinlocks: 8191
              synic:
                enabled: true
              synictimer:
                direct:
                  enabled: true
                enabled: true
              tlbflush:
                enabled: true
              vapic:
                enabled: true
              vpindex:
                enabled: true
            smm:
              enabled: true
          firmware:
            bootloader:
              efi:
                persistent: true
                secureBoot: true
            uuid: 4d20f349-74ac-4035-99e9-c1be74e5c3a1
          machine:
            type: pc-q35-rhel9.6.0
          memory:
            guest: 8Gi
            maxGuest: 32Gi
          resources:
            requests:
              memory: 8Gi
        evictionStrategy: None
        networks:
        - name: default
          pod: {}
        terminationGracePeriodSeconds: 3600
        volumes:
        - dataVolume:
            name: windows-11-dv-mig-l8hs
          name: dv-disk
      status:
        activePods:
          0a7885fc-45d3-4183-b5fb-1240372270b9: cnvqe-075.lab.eng.tlv2.redhat.com
          3a103656-8b4d-4910-af62-c5b8c9188316: cnvqe-075.lab.eng.tlv2.redhat.com
          5d5314bd-69b9-4cb5-8071-c294fcc19911: cnvqe-075.lab.eng.tlv2.redhat.com
          8b20d7be-9f37-43ce-ab8b-7d792c6b3f3f: cnvqe-075.lab.eng.tlv2.redhat.com
          8e82a4d9-dfb2-459c-a85b-68c47167852d: cnvqe-073.lab.eng.tlv2.redhat.com
          9a4d9b40-d085-4985-a339-6376ac818b5d: cnvqe-075.lab.eng.tlv2.redhat.com
          15eb30ee-4479-415d-aed9-6502cb1a720a: cnvqe-073.lab.eng.tlv2.redhat.com
          19c23afb-f7a8-43b4-99dd-91f8a438152e: cnvqe-074.lab.eng.tlv2.redhat.com
          57b89ab5-cd88-48ef-ac0f-814f778802c3: cnvqe-073.lab.eng.tlv2.redhat.com
          75a41eb2-9743-4d29-9ff3-6c506fe958cc: cnvqe-073.lab.eng.tlv2.redhat.com
          85fb818f-f250-40ee-9d84-e037e8b7e971: cnvqe-074.lab.eng.tlv2.redhat.com
          88cac0ad-a345-4045-940e-0455c975d426: cnvqe-074.lab.eng.tlv2.redhat.com
          155e1bd6-b707-4397-b8cc-c722c3910098: cnvqe-073.lab.eng.tlv2.redhat.com
          301daff2-1ac4-47b6-855c-4bbffa3c3b00: cnvqe-074.lab.eng.tlv2.redhat.com
          615ffbb5-8d16-412a-9380-2905ac917498: cnvqe-075.lab.eng.tlv2.redhat.com
          973c224c-e5a7-47cd-bfb3-0981f7ed4aea: cnvqe-075.lab.eng.tlv2.redhat.com
          2294b092-34a7-4312-ab0f-c84628c173a8: cnvqe-075.lab.eng.tlv2.redhat.com
          2647b34f-529b-4223-a13d-be4e404d6935: cnvqe-073.lab.eng.tlv2.redhat.com
          a08fcbd4-ca38-429f-9b00-5f220efc8920: cnvqe-075.lab.eng.tlv2.redhat.com
          ac537eac-e8ba-4778-945d-d464da737c5a: cnvqe-075.lab.eng.tlv2.redhat.com
          b19fe7a7-1faf-4519-a213-bed2221f5671: cnvqe-075.lab.eng.tlv2.redhat.com
          c26c3d74-5c75-4988-b845-9e25091639db: cnvqe-075.lab.eng.tlv2.redhat.com
          dcc7eb72-2ca6-40dc-92e7-01cd91ec3ad9: cnvqe-074.lab.eng.tlv2.redhat.com
          ec680eef-dd4a-4f10-9b5a-618724f3cd20: cnvqe-075.lab.eng.tlv2.redhat.com
          f3ddfaf8-72c0-49d8-9525-e411dd3daca1: cnvqe-074.lab.eng.tlv2.redhat.com
          f23a1b1c-c662-490c-a2df-303ccbb2b5b2: cnvqe-074.lab.eng.tlv2.redhat.com
        conditions:
        - lastProbeTime: null
          lastTransitionTime: "2025-09-07T13:09:23Z"
          status: "True"
          type: Ready
        - lastProbeTime: null
          lastTransitionTime: null
          message: All of the VMI's DVs are bound and ready
          reason: AllDVsReady
          status: "True"
          type: DataVolumesReady
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: StorageLiveMigratable
        - lastProbeTime: "2025-09-07T11:39:42Z"
          lastTransitionTime: null
          status: "True"
          type: AgentConnected
        - lastProbeTime: null
          lastTransitionTime: null
          status: "True"
          type: LiveMigratable
        currentCPUTopology:
          cores: 1
          sockets: 2
          threads: 1
        guestOSInfo:
          id: mswindows
          kernelRelease: "22000"
          kernelVersion: "10.0"
          machine: x86_64
          name: Microsoft Windows
          prettyName: Windows 10 Pro
          version: Microsoft Windows 11
          versionId: "11"
        interfaces:
        - infoSource: domain, guest-agent
          interfaceName: Ethernet Instance 0
          ipAddress: 10.128.1.115
          ipAddresses:
          - 10.128.1.115
          - fd02:0:0:1::4685
          linkState: up
          mac: 0a:58:0a:80:01:51
          name: default
          podInterfaceName: eth0
          queueCount: 1
        - infoSource: guest-agent
          interfaceName: Loopback Pseudo-Interface 1
          ipAddress: 127.0.0.1
          ipAddresses:
          - ::1
          - 127.0.0.1
        - infoSource: guest-agent
          interfaceName: vEthernet (WSL)
          ipAddress: 172.27.16.1
          ipAddresses:
          - fe80::cc4e:88a8:7de8:8228%9
          - 172.27.16.1
          mac: 00:15:5d:dc:45:d3
        launcherContainerImageVersion: registry.redhat.io/container-native-virtualization/virt-launcher-rhel9@sha256:3d619787d670d9b026aae66c8ad0b31a87389ee33165e3e189b71a2b62878e3b
        machine:
          type: pc-q35-rhel9.6.0
        memory:
          guestAtBoot: 8Gi
          guestCurrent: 8Gi
          guestRequested: 8Gi
        migrationMethod: LiveMigration
        migrationState:
          completed: true
          endTimestamp: "2025-09-07T13:09:23Z"
          migrationConfiguration:
            allowAutoConverge: false
            allowPostCopy: false
            allowWorkloadDisruption: false
            bandwidthPerMigration: "0"
            completionTimeoutPerGiB: 150
            nodeDrainTaintKey: kubevirt.io/drain
            parallelMigrationsPerCluster: 5
            parallelOutboundMigrationsPerNode: 2
            progressTimeout: 150
            unsafeMigrationOverride: false
          migrationUid: c0823e5f-09e3-443c-bb4b-ada954ebb5d6
          mode: PreCopy
          sourceNode: cnvqe-073.lab.eng.tlv2.redhat.com
          sourcePersistentStatePVCName: persistent-state-for-windows-11-vm-1757243511-8243139-pksb6
          sourcePod: virt-launcher-windows-11-vm-1757243511-8243139-qvk8m
          startTimestamp: "2025-09-07T13:09:14Z"
          targetDirectMigrationNodePorts:
            "33123": 0
            "34751": 49152
          targetNode: cnvqe-075.lab.eng.tlv2.redhat.com
          targetNodeAddress: 10.128.1.69
          targetNodeDomainDetected: true
          targetNodeDomainReadyTimestamp: "2025-09-07T13:09:23Z"
          targetPersistentStatePVCName: persistent-state-for-windows-11-vm-1757243511-8243139-ttqhp
          targetPod: virt-launcher-windows-11-vm-1757243511-8243139-7tb97
        migrationTransport: Unix
        nodeName: cnvqe-075.lab.eng.tlv2.redhat.com
        phase: Running
        phaseTransitionTimestamps:
        - phase: Pending
          phaseTransitionTimestamp: "2025-09-07T11:11:54Z"
        - phase: Scheduling
          phaseTransitionTimestamp: "2025-09-07T11:11:55Z"
        - phase: Scheduled
          phaseTransitionTimestamp: "2025-09-07T11:39:11Z"
        - phase: Running
          phaseTransitionTimestamp: "2025-09-07T11:39:19Z"
        qosClass: Burstable
        runtimeUser: 107
        selinuxContext: system_u:object_r:container_file_t:s0:c551,c736
        topologyHints:
          tscFrequency: 2095077000
        virtualMachineRevisionName: revision-start-vm-77da8433-9d37-47fa-95ea-064d55ad49f8-2
        volumeStatus:
        - name: dv-disk
          persistentVolumeClaimInfo:
            accessModes:
            - ReadWriteMany
            capacity:
              storage: 70Gi
            claimName: windows-11-dv-mig-l8hs
            filesystemOverhead: "0"
            requests:
              storage: "75161927680"
            volumeMode: Block
          target: vda
        - name: persistent-state-for-windows-11-vm-1757243511-8243139-ttqhp
          persistentVolumeClaimInfo:
            accessModes:
            - ReadWriteOnce
            claimName: persistent-state-for-windows-11-vm-1757243511-8243139-ttqhp
          target: ""
       

              akalenyu Alex Kalenyuk
              jpeimer@redhat.com Jenia Peimer
              Jenia Peimer Jenia Peimer
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: