-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
2.6.4
-
None
-
False
-
None
-
False
-
-
-
Important
Hello,
We have a issue where Forklift pod is crashing frequently from openshift-mtv namespace.
$ oc get pods -n openshift-mtv
NAME READY STATUS RESTARTS AGE
forklift-api-85487cf4bc-sppcm 1/1 Running 0 8h
forklift-controller-b7f544cdf-jm46p 1/2 CrashLoopBackOff 14 (13s ago) 5h14m
forklift-operator-778b6b9d87-6ljkw 1/1 Running 0 8h
forklift-ui-plugin-6856fc86fc-zbnkr 1/1 Running 0 8h
forklift-validation-5fcb4c7bc5-jqqk9 1/1 Running 0 8h
forklift-volume-populator-controller-66dbd8dffc-xj85l 1/1 Running 0 8h
Inventory container logs show:
2024-09-02T11:38:00.030851409Z {"level":"error","ts":"2024-09-02 11:38:00.030","logger":"collector|vsphere","msg":"full queue, event discarded","provider":"vms-poc/vspehere-hqidlinfvs51","error":"full queue, event discarded","stacktrace":"github.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Error\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:385\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).notify\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:213\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Journal).Report\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:355\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).report\n\t/remote-source/app/pkg/lib/inventory/model/client.go:681\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit.func1\n\t/remote-source/app/pkg/lib/inventory/model/client.go:595\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit\n\t/remote-source/app/pkg/lib/inventory/model/client.go:611\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:457\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:399\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Updated\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:379\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).Start.func1\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:265"}
2024-09-02T11:38:00.114743951Z {"level":"error","ts":"2024-09-02 11:38:00.114","logger":"collector|vsphere","msg":"full queue, event discarded","provider":"vms-poc/vspehere-hqidlinfvs51","error":"full queue, event discarded","stacktrace":"github.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Error\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:385\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).notify\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:213\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Journal).Report\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:355\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).report\n\t/remote-source/app/pkg/lib/inventory/model/client.go:681\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit.func1\n\t/remote-source/app/pkg/lib/inventory/model/client.go:595\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit\n\t/remote-source/app/pkg/lib/inventory/model/client.go:611\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:457\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).Updated\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:417\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).Start.func1\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:265"}
We notice that the Main container pod inside forklift is is getting runtime error: invalid memory address or nil pointer dereference
2024-09-02T11:34:46.531339970Z {"level":"info","ts":"2024-09-02 11:34:46.531","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"plan","object":{"name":"test-migration-vm-9-migration-2","namespace":"openshift-mtv"},"namespace":"openshift-mtv","name":"test-migration-vm-9-migration-2","reconcileID":"bd4459a0-89c2-4752-9821-b8b0dbc8d85d"}
2024-09-02T11:34:46.533466002Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2024-09-02T11:34:46.533466002Z panic: runtime error: invalid memory address or nil pointer dereference
2024-09-02T11:34:46.533466002Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d34da2]
2024-09-02T11:34:46.533473974Z
2024-09-02T11:34:46.533473974Z goroutine 271 [running]:
2024-09-02T11:34:46.533482847Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2024-09-02T11:34:46.533482847Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
2024-09-02T11:34:46.533486660Z panic({0x2c39780?, 0x4fa1b70?})
2024-09-02T11:34:46.533490313Z /usr/lib/golang/src/runtime/panic.go:914 +0x21f
2024-09-02T11:34:46.533504419Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).mapDisks(0xc004b82fa8, 0x2cfd4c0?, {{0xc000f99d40, 0xa}, {0xc0008babd0, 0x13}, {0x0, 0x0}, {0x0, 0x0}}, ...)
2024-09-02T11:34:46.533514447Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:706 +0x542
2024-09-02T11:34:46.533517733Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).VirtualMachine(0xc004b82fa8, {{0xc000f99d40, 0xa}, {0xc0008babd0, 0x13}, {0x0, 0x0}, {0x0, 0x0}}, 0xc004931308, ...)
2024-09-02T11:34:46.533527717Z /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:534 +0x485
2024-09-02T11:34:46.533535430Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).virtualMachine(0xc000675f98, 0xc000fa0ea0)
2024-09-02T11:34:46.533538693Z /remote-source/app/pkg/controller/plan/kubevirt.go:1275 +0xaba
2024-09-02T11:34:46.533542288Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).EnsureVM(0xc000675f98, 0xc000fa0ea0)
2024-09-02T11:34:46.533542288Z /remote-source/app/pkg/controller/plan/kubevirt.go:389 +0x39a
2024-09-02T11:34:46.533611820Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).execute(0xc000675f80, 0xc000fa0ea0)
2024-09-02T11:34:46.533611820Z /remote-source/app/pkg/controller/plan/migration.go:825 +0x17ce5
2024-09-02T11:34:46.533615427Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).Run(0xc000675f80)
2024-09-02T11:34:46.533615427Z /remote-source/app/pkg/controller/plan/migration.go:195 +0x285
2024-09-02T11:34:46.533622255Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Reconciler).execute(0xc003dd2c60, 0xc000e45c00)
2024-09-02T11:34:46.533622255Z /remote-source/app/pkg/controller/plan/controller.go:428 +0x98b
2024-09-02T11:34:46.533626261Z github.com/konveyor/forklift-controller/pkg/controller/plan.Reconciler.Reconcile({{{0x3697178, 0xc000177700}, {0x36b2f40, 0xc0008da7e0}, {0x36a9c78, 0xc00016b5e0}}}, {0x36c23a0?, 0xc000c4dce8?}, {{{0xc000363ce0, 0xd}, ...}})
2024-09-02T11:34:46.533639832Z /remote-source/app/pkg/controller/plan/controller.go:257 +0x9de
2024-09-02T11:34:46.533643352Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x36a7f48?, {0x36a4838?, 0xc003dd2c30?}, {{{0xc000363ce0?, 0xb?}, {0xc00057cb20?, 0x0?}}})
2024-09-02T11:34:46.533653431Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
2024-09-02T11:34:46.533656821Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006b5cc0, {0x36a4870, 0xc00075b6d0}, {0x2e5f7a0?, 0xc000c04260?})
2024-09-02T11:34:46.533663573Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
2024-09-02T11:34:46.533667045Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006b5cc0, {0x36a4870, 0xc00075b6d0})
2024-09-02T11:34:46.533670539Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
2024-09-02T11:34:46.533673897Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
2024-09-02T11:34:46.533673897Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
2024-09-02T11:34:46.533677381Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 187
2024-09-02T11:34:46.533677381Z /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565
The pod memory usage:
# oc adm top pods -n openshift-mtv
NAME CPU(cores) MEMORY(bytes)
forklift-api-85487cf4bc-sppcm 0m 33Mi
forklift-controller-b7f544cdf-jm46p 925m 927Mi <===
forklift-operator-778b6b9d87-6ljkw 0m 129Mi
forklift-ui-plugin-6856fc86fc-zbnkr 0m 2Mi
forklift-validation-5fcb4c7bc5-jqqk9 221m 119Mi
forklift-volume-populator-controller-66dbd8dffc-xj85l 1m 116Mi
Version:
Migration Tool Kit for Virtualization: "version":"2.6.4"