Description of problem:
During a virtual machine migration, the forklift control pod enters a Error/CrashLoopBackOff state.
In the Main container logs we see the following panic:
~~~
2025-03-03T19:15:31.628724969Z {"level":"info","ts":"2025-03-03 19:15:31.628","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"plan","object":{"name":"plan_name","namespace":"openshift-mtv"},"namespace":"openshift-mtv","name":"plan_name","reconcileID":"0ffbfc41-ce7d-1111-2222-f1f84f5bb2f9"}
2025-03-03T19:15:31.630841857Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2025-03-03T19:15:31.630841857Z panic: runtime error: invalid memory address or nil pointer dereference
2025-03-03T19:15:31.630841857Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d49d09]
2025-03-03T19:15:31.630849775Z
2025-03-03T19:15:31.630849775Z goroutine 400 [running]:
2025-03-03T19:15:31.630859246Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2025-03-03T19:15:31.630859246Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
2025-03-03T19:15:31.630863161Z panic({0x2c3e700?, 0x4faeb40?})
2025-03-03T19:15:31.630866846Z /usr/lib/golang/src/runtime/panic.go:914 +0x21f
2025-03-03T19:15:31.630871653Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).mapDisks(0xc0036052d8, 0x2d028c0?, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, ...)
2025-03-03T19:15:31.630882045Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:714 +0x549
2025-03-03T19:15:31.630885791Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).VirtualMachine(0xc0036052d8, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, 0xc00132db08, ...)
2025-03-03T19:15:31.630895984Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:545 +0x485
2025-03-03T19:15:31.630899548Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).virtualMachine(0xc003295eb8, 0xc001d02c40)
2025-03-03T19:15:31.630902930Z /remote-source/app/pkg/controller/plan/kubevirt.go:1353 +0xab3
2025-03-03T19:15:31.630906571Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).EnsureVM(0xc003295eb8, 0xc001d02c40)
2025-03-03T19:15:31.630906571Z /remote-source/app/pkg/controller/plan/kubevirt.go:436 +0x39a
2025-03-03T19:15:31.630940319Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).execute(0xc003295ea0, 0xc001d02c40)
2025-03-03T19:15:31.630940319Z /remote-source/app/pkg/controller/plan/migration.go:890 +0xade8
2025-03-03T19:15:31.630944138Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).Run(0xc003295ea0)
2025-03-03T19:15:31.630944138Z /remote-source/app/pkg/controller/plan/migration.go:219 +0x285
2025-03-03T19:15:31.630948033Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Reconciler).execute(0xc001d91590, 0xc001b57800)
2025-03-03T19:15:31.630951539Z /remote-source/app/pkg/controller/plan/controller.go:428 +0x98b
2025-03-03T19:15:31.630955068Z github.com/konveyor/forklift-controller/pkg/controller/plan.Reconciler.Reconcile({{{0x36a1518, 0xc00089af80}, {0x36bd2c0, 0xc0007c8510}, {0x36b4018, 0xc0007b9d60}}}, {0x36cb840?, 0xc000ac9ce8?}, {{{0xc002a89650, 0xd}, ...}})
2025-03-03T19:15:31.630968510Z /remote-source/app/pkg/controller/plan/controller.go:257 +0x9de
2025-03-03T19:15:31.630972006Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x36b22e8?, {0x36aebd8?, 0xc001d91560?}, {{{0xc002a89650?, 0xb?}, {0xc002314210?, 0x0?}}})
2025-03-03T19:15:31.630978590Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
2025-03-03T19:15:31.630982031Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000d17360, {0x36aec10, 0xc0007b7540}, {0x2e657c0?, 0xc001414100?})
2025-03-03T19:15:31.630988862Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
2025-03-03T19:15:31.630992362Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000d17360, {0x36aec10, 0xc0007b7540})
2025-03-03T19:15:31.630992362Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
2025-03-03T19:15:31.630995977Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
2025-03-03T19:15:31.630995977Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
2025-03-03T19:15:31.630999768Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 174
2025-03-03T19:15:31.631003224Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565
~~~
Version-Release number of selected component (if applicable):
OpenShift: 4.16.5 CNV: 4.16.6 MTV: v2.7.10
How reproducible:
Unknown
Steps to Reproduce:
1. Create multi virtual machine migration plan from VMware provider. 2. Start the migration
Actual results:
Some virtual machines are successfully migrated, others fail to complete due to crashing forklift control pod.
Expected results:
All virtual machines successfully migrate.
Additional info:
The current workaround is to cancel the migration, archive the plan and create a new plan with the virtual machines that didn't migrate during the first plan.