Description of problem:
During a virtual machine migration, the forklift control pod enters a Error/CrashLoopBackOff state. In the Main container logs we see the following panic: ~~~ 2025-03-03T19:15:31.628724969Z {"level":"info","ts":"2025-03-03 19:15:31.628","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"plan","object":{"name":"plan_name","namespace":"openshift-mtv"},"namespace":"openshift-mtv","name":"plan_name","reconcileID":"0ffbfc41-ce7d-1111-2222-f1f84f5bb2f9"} 2025-03-03T19:15:31.630841857Z panic: runtime error: invalid memory address or nil pointer dereference [recovered] 2025-03-03T19:15:31.630841857Z panic: runtime error: invalid memory address or nil pointer dereference 2025-03-03T19:15:31.630841857Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d49d09] 2025-03-03T19:15:31.630849775Z 2025-03-03T19:15:31.630849775Z goroutine 400 [running]: 2025-03-03T19:15:31.630859246Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() 2025-03-03T19:15:31.630859246Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5 2025-03-03T19:15:31.630863161Z panic({0x2c3e700?, 0x4faeb40?}) 2025-03-03T19:15:31.630866846Z /usr/lib/golang/src/runtime/panic.go:914 +0x21f 2025-03-03T19:15:31.630871653Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).mapDisks(0xc0036052d8, 0x2d028c0?, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, ...) 2025-03-03T19:15:31.630882045Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:714 +0x549 2025-03-03T19:15:31.630885791Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).VirtualMachine(0xc0036052d8, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, 0xc00132db08, ...) 2025-03-03T19:15:31.630895984Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:545 +0x485 2025-03-03T19:15:31.630899548Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).virtualMachine(0xc003295eb8, 0xc001d02c40) 2025-03-03T19:15:31.630902930Z /remote-source/app/pkg/controller/plan/kubevirt.go:1353 +0xab3 2025-03-03T19:15:31.630906571Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).EnsureVM(0xc003295eb8, 0xc001d02c40) 2025-03-03T19:15:31.630906571Z /remote-source/app/pkg/controller/plan/kubevirt.go:436 +0x39a 2025-03-03T19:15:31.630940319Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).execute(0xc003295ea0, 0xc001d02c40) 2025-03-03T19:15:31.630940319Z /remote-source/app/pkg/controller/plan/migration.go:890 +0xade8 2025-03-03T19:15:31.630944138Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).Run(0xc003295ea0) 2025-03-03T19:15:31.630944138Z /remote-source/app/pkg/controller/plan/migration.go:219 +0x285 2025-03-03T19:15:31.630948033Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Reconciler).execute(0xc001d91590, 0xc001b57800) 2025-03-03T19:15:31.630951539Z /remote-source/app/pkg/controller/plan/controller.go:428 +0x98b 2025-03-03T19:15:31.630955068Z github.com/konveyor/forklift-controller/pkg/controller/plan.Reconciler.Reconcile({{{0x36a1518, 0xc00089af80}, {0x36bd2c0, 0xc0007c8510}, {0x36b4018, 0xc0007b9d60}}}, {0x36cb840?, 0xc000ac9ce8?}, {{{0xc002a89650, 0xd}, ...}}) 2025-03-03T19:15:31.630968510Z /remote-source/app/pkg/controller/plan/controller.go:257 +0x9de 2025-03-03T19:15:31.630972006Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x36b22e8?, {0x36aebd8?, 0xc001d91560?}, {{{0xc002a89650?, 0xb?}, {0xc002314210?, 0x0?}}}) 2025-03-03T19:15:31.630978590Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7 2025-03-03T19:15:31.630982031Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000d17360, {0x36aec10, 0xc0007b7540}, {0x2e657c0?, 0xc001414100?}) 2025-03-03T19:15:31.630988862Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc 2025-03-03T19:15:31.630992362Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000d17360, {0x36aec10, 0xc0007b7540}) 2025-03-03T19:15:31.630992362Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9 2025-03-03T19:15:31.630995977Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() 2025-03-03T19:15:31.630995977Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79 2025-03-03T19:15:31.630999768Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 174 2025-03-03T19:15:31.631003224Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565 ~~~
Version-Release number of selected component (if applicable):
OpenShift: 4.16.5 CNV: 4.16.6 MTV: v2.7.10
How reproducible:
Unknown
Steps to Reproduce:
1. Create multi virtual machine migration plan from VMware provider. 2. Start the migration
Actual results:
Some virtual machines are successfully migrated, others fail to complete due to crashing forklift control pod.
Expected results:
All virtual machines successfully migrate.
Additional info:
The current workaround is to cancel the migration, archive the plan and create a new plan with the virtual machines that didn't migrate during the first plan.