VM snapshot fails to restore

    Openshift Virtualization 4.18.3

    1. Take a snapshot of the affected VM using web interface
    2. Wait a couple days
    3. Poweroff the VM
    4. Try to restore the snapshot
    

    Snapshot restore never completes

    Snapshot is restore successfully

    Not sure if it matters, but the VM had been previously migrated from one storage (HPE Alletra) to another storage (ODF) using Openshift's Migration. HPE storage uses HPE CSI driver. Migration completed successfully as far as we know.

Here is the VM's yaml:

sh-5.1# oc get vm rhv-manager -oyaml -n test-vms
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  annotations:
    kubemacpool.io/transaction-timestamp: "2025-06-23T17:47:00.34952844Z"
    kubevirt.io/latest-observed-api-version: v1
    kubevirt.io/storage-observed-api-version: v1
    migrate-storage-class/rhv-manager-mig-fw4pwg: rhv-manager
    vm.kubevirt.io/validations: |
      [
        {
          "name": "minimal-required-memory",
          "path": "jsonpath::.spec.domain.memory.guest",
          "rule": "integer",
          "message": "This VM requires more memory.",
          "min": 1610612736
        }
      ]
  creationTimestamp: "2025-06-20T17:08:45Z"
  finalizers:
  - kubevirt.io/virtualMachineControllerFinalize
  generation: 12
  labels:
    app: rhv-manager
    vm.kubevirt.io/template: rhel8-server-small
    vm.kubevirt.io/template.namespace: openshift
    vm.kubevirt.io/template.revision: "1"
    vm.kubevirt.io/template.version: v0.32.2
  name: rhv-manager
  namespace: test-vms
  resourceVersion: "15788627"
  uid: 367debab-6cef-4bf3-99a6-6bfc465b5e38
spec:
  dataVolumeTemplates:
  - apiVersion: cdi.kubevirt.io/v1beta1
    kind: DataVolume
    metadata:
      creationTimestamp: null
      name: rhv-manager-mig-fw4pwg
    spec:
      source:
        pvc:
          name: rhv-manager
          namespace: test-vms
      storage:
        resources:
          requests:
            storage: "322122547200"
        storageClassName: ocs-storagecluster-ceph-rbd-virtualization
  runStrategy: Halted
  template:
    metadata:
      annotations:
        vm.kubevirt.io/flavor: small
        vm.kubevirt.io/os: rhel8
        vm.kubevirt.io/workload: server
      creationTimestamp: null
      labels:
        kubevirt.io/domain: rhv-manager
        kubevirt.io/size: small
        network.kubevirt.io/headlessService: headless
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 1
          sockets: 2
          threads: 1
        devices:
          disks:
          - bootOrder: 1
            disk:
              bus: sata
            name: rootdisk
          interfaces:
          - bridge: {}
            macAddress: 02:6c:b5:00:00:01
            model: virtio
            name: nic-01
          logSerialConsole: false
          rng: {}
        machine:
          type: pc-q35-rhel9.4.0
        memory:
          guest: 16Gi
        resources: {}
      networks:
      - multus:
          networkName: default/vlan4
        name: nic-01
      terminationGracePeriodSeconds: 180
      volumes:
      - dataVolume:
          name: rhv-manager-mig-fw4pwg
        name: rootdisk
  updateVolumesStrategy: Migration
status:
  conditions:
  - lastProbeTime: "2025-06-23T17:47:26Z"
    lastTransitionTime: "2025-06-23T17:47:26Z"
    message: VMI does not exist
    reason: VMINotExists
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: null
    message: All of the VMI's DVs are bound and not running
    reason: AllDVsReady
    status: "True"
    type: DataVolumesReady
  - lastProbeTime: null
    lastTransitionTime: null
    status: "True"
    type: LiveMigratable
  - lastProbeTime: null
    lastTransitionTime: null
    status: "True"
    type: StorageLiveMigratable
  desiredGeneration: 12
  observedGeneration: 12
  printableStatus: Stopped
  restoreInProgress: restore-rhv-manager-snapshot-20250620-163822-1750702545908
  runStrategy: Halted
  volumeSnapshotStatuses:
  - enabled: true
    name: rootdisk
  volumeUpdateState:
    volumeMigrationState:
      migratedVolumes:
      - destinationPVCInfo:
          claimName: rhv-manager-mig-fw4pwg
          volumeMode: Block
        sourcePVCInfo:
          claimName: rhv-manager
          volumeMode: Block
        volumeName: rootdisk
sh-5.1# 


And here's the vmrestore object:


sh-5.1# oc get vmrestore restore-rhv-manager-snapshot-20250620-163822-1750702545908 -n test-vms -oyaml
apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineRestore
metadata:
  creationTimestamp: "2025-06-23T18:15:50Z"
  finalizers:
  - snapshot.kubevirt.io/vmrestore-protection
  generation: 1
  name: restore-rhv-manager-snapshot-20250620-163822-1750702545908
  namespace: test-vms
  ownerReferences:
  - apiVersion: kubevirt.io/v1
    blockOwnerDeletion: false
    kind: VirtualMachine
    name: rhv-manager
    uid: 367debab-6cef-4bf3-99a6-6bfc465b5e38
  resourceVersion: "15788647"
  uid: e909a4ca-4aea-4e2e-b197-5ee43d6b66c7
spec:
  target:
    apiGroup: kubevirt.io
    kind: VirtualMachine
    name: rhv-manager
  virtualMachineSnapshotName: rhv-manager-snapshot-20250620-163822
status:
  complete: false
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-06-23T18:15:51Z"
    reason: 'admission webhook "virtualmachine-validator.kubevirt.io" denied the request:
      DataVolumeTemplate entry spec.dataVolumeTemplate[0] must be referenced in the
      VMI template''s ''volumes'' list'
    status: "False"
    type: Progressing
  - lastProbeTime: null
    lastTransitionTime: "2025-06-23T18:15:51Z"
    reason: 'admission webhook "virtualmachine-validator.kubevirt.io" denied the request:
      DataVolumeTemplate entry spec.dataVolumeTemplate[0] must be referenced in the
      VMI template''s ''volumes'' list'
    status: "False"
    type: Ready
  deletedDataVolumes:
  - rhv-manager-mig-fw4pwg
  restores:
  - persistentVolumeClaim: restore-e909a4ca-4aea-4e2e-b197-5ee43d6b66c7-rootdisk
    volumeName: rootdisk
    volumeSnapshotName: vmsnapshot-5da734e6-7629-428e-8540-363ba1676773-volume-rootdisk
sh-5.1# 


So far, customer has not been able to restore the VM.