Uploaded image for project: 'Migration Toolkit for Virtualization'
  1. Migration Toolkit for Virtualization
  2. MTV-2164

Forklift controller Main container crashing during migration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • 2.7.11
    • 2.7.10
    • Guest
    • Incidents & Support
    • 3
    • False
    • Hide

      None

      Show
      None
    • True
    • Important
    • Customer Reported

      Description of problem:

      During a virtual machine migration, the forklift control pod enters a Error/CrashLoopBackOff state.
      
      In the Main container logs we see the following panic:
      
      ~~~
      2025-03-03T19:15:31.628724969Z {"level":"info","ts":"2025-03-03 19:15:31.628","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"plan","object":{"name":"plan_name","namespace":"openshift-mtv"},"namespace":"openshift-mtv","name":"plan_name","reconcileID":"0ffbfc41-ce7d-1111-2222-f1f84f5bb2f9"}
      2025-03-03T19:15:31.630841857Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      2025-03-03T19:15:31.630841857Z  panic: runtime error: invalid memory address or nil pointer dereference
      2025-03-03T19:15:31.630841857Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d49d09]
      2025-03-03T19:15:31.630849775Z 
      2025-03-03T19:15:31.630849775Z goroutine 400 [running]:
      2025-03-03T19:15:31.630859246Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
      2025-03-03T19:15:31.630859246Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
      2025-03-03T19:15:31.630863161Z panic({0x2c3e700?, 0x4faeb40?})
      2025-03-03T19:15:31.630866846Z  /usr/lib/golang/src/runtime/panic.go:914 +0x21f
      2025-03-03T19:15:31.630871653Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).mapDisks(0xc0036052d8, 0x2d028c0?, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, ...)
      2025-03-03T19:15:31.630882045Z  /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:714 +0x549
      2025-03-03T19:15:31.630885791Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).VirtualMachine(0xc0036052d8, {{0xc0018416c0, 0xa}, {0xc0018416d0, 0xe}, {0x0, 0x0}, {0x0, 0x0}}, 0xc00132db08, ...)
      2025-03-03T19:15:31.630895984Z  /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:545 +0x485
      2025-03-03T19:15:31.630899548Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).virtualMachine(0xc003295eb8, 0xc001d02c40)
      2025-03-03T19:15:31.630902930Z  /remote-source/app/pkg/controller/plan/kubevirt.go:1353 +0xab3
      2025-03-03T19:15:31.630906571Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).EnsureVM(0xc003295eb8, 0xc001d02c40)
      2025-03-03T19:15:31.630906571Z  /remote-source/app/pkg/controller/plan/kubevirt.go:436 +0x39a
      2025-03-03T19:15:31.630940319Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).execute(0xc003295ea0, 0xc001d02c40)
      2025-03-03T19:15:31.630940319Z  /remote-source/app/pkg/controller/plan/migration.go:890 +0xade8
      2025-03-03T19:15:31.630944138Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).Run(0xc003295ea0)
      2025-03-03T19:15:31.630944138Z  /remote-source/app/pkg/controller/plan/migration.go:219 +0x285
      2025-03-03T19:15:31.630948033Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Reconciler).execute(0xc001d91590, 0xc001b57800)
      2025-03-03T19:15:31.630951539Z  /remote-source/app/pkg/controller/plan/controller.go:428 +0x98b
      2025-03-03T19:15:31.630955068Z github.com/konveyor/forklift-controller/pkg/controller/plan.Reconciler.Reconcile({{{0x36a1518, 0xc00089af80}, {0x36bd2c0, 0xc0007c8510}, {0x36b4018, 0xc0007b9d60}}}, {0x36cb840?, 0xc000ac9ce8?}, {{{0xc002a89650, 0xd}, ...}})
      2025-03-03T19:15:31.630968510Z  /remote-source/app/pkg/controller/plan/controller.go:257 +0x9de
      2025-03-03T19:15:31.630972006Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x36b22e8?, {0x36aebd8?, 0xc001d91560?}, {{{0xc002a89650?, 0xb?}, {0xc002314210?, 0x0?}}})
      2025-03-03T19:15:31.630978590Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
      2025-03-03T19:15:31.630982031Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000d17360, {0x36aec10, 0xc0007b7540}, {0x2e657c0?, 0xc001414100?})
      2025-03-03T19:15:31.630988862Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
      2025-03-03T19:15:31.630992362Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000d17360, {0x36aec10, 0xc0007b7540})
      2025-03-03T19:15:31.630992362Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
      2025-03-03T19:15:31.630995977Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
      2025-03-03T19:15:31.630995977Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
      2025-03-03T19:15:31.630999768Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 174
      2025-03-03T19:15:31.631003224Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565
      ~~~

      Version-Release number of selected component (if applicable):

      OpenShift: 4.16.5
      CNV: 4.16.6
      MTV: v2.7.10

      How reproducible:

      Unknown

      Steps to Reproduce:

      1. Create multi virtual machine migration plan from VMware provider.
      2. Start the migration 

      Actual results:

      Some virtual machines are successfully migrated, others fail to complete due to crashing forklift control pod.

      Expected results:

      All virtual machines successfully migrate.

      Additional info:

      The current workaround is to cancel the migration, archive the plan and create a new plan with the virtual machines that didn't migrate during the first plan.

              mnecas@redhat.com Martin Necas
              rh-support-jcutler Jonathan Cutler
              Chenli Hu Chenli Hu
              Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: