-
Bug
-
Resolution: Done
-
Major
-
None
-
2.10.1
-
Quality / Stability / Reliability
-
False
-
-
False
-
-
Storage-Offload migration fails during the conversion phase because the virt-v2v and pvcinit pods cannot pull the required image from registry.redhat.io.
Both pods remain in ImagePullBackOff, and the migration never progresses past the conversion step.
the populate pod completed successfully
Environment
- MTV Version tested: 2.10.1-2
- single VM 50GB ISCSI DS VMware 7.0.3
- Platform: OpenShift 4.18.2
- Migration type: Storage-Offload
- Storage backend: NetApp iSCSI (ONTAP)
- ESXi source: vSphere environment (lab)
{
"feature_copy_offload": "true",
"feature_ui_plugin": "true",
"feature_validation": "true",
"feature_volume_populator": "true"
}
[kni@f01-h07-000-r640 ~]$ oc get pods -noffload NAME READY STATUS RESTARTS AGE offload-vm-37005-8fhss 0/1 ImagePullBackOff 0 9m15s offload-vm-37005-pvcinit-g8hwq 0/1 ImagePullBackOff 0 11m populate-03314288-5441-4d15-8315-a51c8f87beae 0/1 Completed 0 11m
[kni@f01-h07-000-r640 ~]$ oc describe pods/offload-vm-37005-8fhss -noffload Name: offload-vm-37005-8fhss Namespace: offload Priority: 0 Service Account: default Node: worker000-r640/10.1.60.13 Start Time: Tue, 25 Nov 2025 09:04:10 +0000 Labels: forklift.app=virt-v2v migration=820025b2-8f6d-40f8-ae61-868254c8d188 plan=7f1a9860-4bf8-4cfb-849b-f80ec2676f74 resource=vm-config vmID=vm-37005 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.130.0.182/23"],"mac_address":"0a:58:0a:82:00:b6","gateway_ips":["10.130.0.1"],"routes":[{"dest":"10.128.0.... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.130.0.182" ], "mac": "0a:58:0a:82:00:b6", "default": true, "dns": {} }] openshift.io/scc: forklift-controller-scc seccomp.security.alpha.kubernetes.io/pod: localhost/profiles/unshare.json Status: Pending IP: 10.130.0.182 IPs: IP: 10.130.0.182 Containers: virt-v2v: Container ID: Image: registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 Image ID: Port: 2112/TCP Host Port: 0/TCP State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Limits: cpu: 4 devices.kubevirt.io/kvm: 1 memory: 8Gi Requests: cpu: 1 devices.kubevirt.io/kvm: 1 memory: 1Gi Environment Variables from: offload-vm-37005-frn97 Secret with prefix 'V2V_' Optional: false Environment: V2V_preserveStaticIPs: true V2V_vmName: offload-50gb-iscsi V2V_libvirtURL: esx://root@172.16.11.3?no_verify=1 V2V_source: vSphere V2V_fingerprint: 29:DD:F6:1E:E5:9F:DA:98:44:FA:18:72:60:39:22:E6:8D:C0:5B:22 V2V_extra_args: [] V2V_inPlace: 1 LOCAL_MIGRATION: true Mounts: /mnt/v2v from libvirt-domain-xml (rw) /opt from vddk-vol-mount (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-wt9sq (ro) Devices: /dev/block0 from offl-offl-disk-0-bf0790a9 Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: offl-offl-disk-0-bf0790a9: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: offl-offl-disk-0-bf0790a9 ReadOnly: false libvirt-domain-xml: Type: ConfigMap (a volume populated by a ConfigMap) Name: offload-vm-37005-cpkxm Optional: false vddk-vol-mount: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-wt9sq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: kubevirt.io/schedulable=true Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 95m default-scheduler Successfully assigned offload/offload-vm-37005-8fhss to worker000-r640 Normal SuccessfulAttachVolume 95m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-2d62457b-526e-483f-bfc1-f72f074a762c" Normal SuccessfulMountVolume 95m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-2d62457b-526e-483f-bfc1-f72f074a762c" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-2d62457b-526e-483f-bfc1-f72f074a762c/dev" Normal SuccessfulMountVolume 95m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-2d62457b-526e-483f-bfc1-f72f074a762c" volumeMapPath "/var/lib/kubelet/pods/e7b806bb-5ce7-4cb9-b199-453aa97730d1/volumeDevices/kubernetes.io~csi" Normal AddedInterface 95m multus Add eth0 [10.130.0.182/23] from ovn-kubernetes Normal Pulling 93m (x4 over 95m) kubelet Pulling image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505" Warning Failed 93m (x4 over 95m) kubelet Failed to pull image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505": initializing source docker://registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: (Mirrors also failed: [brew.registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: reading manifest sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 in brew.registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10: name unknown: Digest not found]): registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: reading manifest sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 in registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10: unauthorized: access to the requested resource is not authorized Warning Failed 93m (x4 over 95m) kubelet Error: ErrImagePull Warning Failed 93m (x5 over 94m) kubelet Error: ImagePullBackOff Normal BackOff 4m52s (x394 over 94m) kubelet Back-off pulling image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505"
[kni@f01-h07-000-r640 ~]$ oc describe pods/offload-vm-37005-pvcinit-g8hwq -noffload Name: offload-vm-37005-pvcinit-g8hwq Namespace: offload Priority: 0 Service Account: default Node: worker004-r640/10.1.60.17 Start Time: Tue, 25 Nov 2025 09:04:14 +0000 Labels: forklift.app=consumer migration=820025b2-8f6d-40f8-ae61-868254c8d188 plan=7f1a9860-4bf8-4cfb-849b-f80ec2676f74 resource=vm-config vmID=vm-37005 Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.0.193/23"],"mac_address":"0a:58:0a:81:00:c1","gateway_ips":["10.129.0.1"],"routes":[{"dest":"10.128.0.... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.129.0.193" ], "mac": "0a:58:0a:81:00:c1", "default": true, "dns": {} }] openshift.io/scc: forklift-controller-scc seccomp.security.alpha.kubernetes.io/pod: runtime/default Status: Pending IP: 10.129.0.193 IPs: IP: 10.129.0.193 Containers: main: Container ID: Image: registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 Image ID: Port: <none> Host Port: <none> Command: /bin/sh State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Limits: cpu: 4 devices.kubevirt.io/kvm: 1 memory: 8Gi Requests: cpu: 1 devices.kubevirt.io/kvm: 1 memory: 1Gi Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qmjxv (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: offl-offl-disk-0-bf0790a9: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: offl-offl-disk-0-bf0790a9 ReadOnly: false kube-api-access-qmjxv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: Burstable Node-Selectors: kubevirt.io/schedulable=true Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 95m (x5 over 98m) default-scheduler 0/9 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/9 nodes are available: 9 Preemption is not helpful for scheduling. Normal Scheduled 95m default-scheduler Successfully assigned offload/offload-vm-37005-pvcinit-g8hwq to worker004-r640 Normal AddedInterface 95m multus Add eth0 [10.129.0.193/23] from ovn-kubernetes Normal SuccessfulAttachVolume 95m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-2d62457b-526e-483f-bfc1-f72f074a762c" Normal Pulling 94m (x4 over 95m) kubelet Pulling image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505" Warning Failed 94m (x4 over 95m) kubelet Failed to pull image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505": initializing source docker://registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: (Mirrors also failed: [brew.registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: reading manifest sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 in brew.registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10: name unknown: Digest not found]): registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505: reading manifest sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505 in registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10: unauthorized: access to the requested resource is not authorized Warning Failed 94m (x4 over 95m) kubelet Error: ErrImagePull Warning Failed 93m (x6 over 95m) kubelet Error: ImagePullBackOff Normal BackOff 20s (x416 over 95m) kubelet Back-off pulling image "registry.redhat.io/migration-toolkit-virtualization/mtv-virt-v2v-rhel10@sha256:f666469d0d39368c743ab5410a87b6fed4a48280d6510f47501cde1e6c83a505"
[kni@f01-h07-000-r640 ~]$ oc get pvc -noffload NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE offl-offl-disk-0-bf0790a9 Bound pvc-2d62457b-526e-483f-bfc1-f72f074a762c 50Gi RWX netapp-ontap-perf-iscsi <unset> 98m prime-03314288-5441-4d15-8315-a51c8f87beae Terminating pvc-2d62457b-526e-483f-bfc1-f72f074a762c 0 netapp-ontap-perf-iscsi <unset> 98m
