Loading...

Type: Bug
Resolution: Done
Priority: Normal
Fix Version/s: None
Affects Version/s: 2.6.4
Component/s: Controller
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

Severity:
Important

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

Hello,

We have a issue where Forklift pod is crashing frequently from openshift-mtv namespace.
$ oc get pods -n openshift-mtv
NAME READY STATUS RESTARTS AGE
forklift-api-85487cf4bc-sppcm 1/1 Running 0 8h
forklift-controller-b7f544cdf-jm46p 1/2 CrashLoopBackOff 14 (13s ago) 5h14m
forklift-operator-778b6b9d87-6ljkw 1/1 Running 0 8h
forklift-ui-plugin-6856fc86fc-zbnkr 1/1 Running 0 8h
forklift-validation-5fcb4c7bc5-jqqk9 1/1 Running 0 8h
forklift-volume-populator-controller-66dbd8dffc-xj85l 1/1 Running 0 8h
Inventory container logs show:

2024-09-02T11:38:00.030851409Z {"level":"error","ts":"2024-09-02 11:38:00.030","logger":"collector|vsphere","msg":"full queue, event discarded","provider":"vms-poc/vspehere-hqidlinfvs51","error":"full queue, event discarded","stacktrace":"github.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Error\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:385\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).notify\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:213\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Journal).Report\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:355\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).report\n\t/remote-source/app/pkg/lib/inventory/model/client.go:681\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit.func1\n\t/remote-source/app/pkg/lib/inventory/model/client.go:595\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit\n\t/remote-source/app/pkg/lib/inventory/model/client.go:611\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:457\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:399\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Updated\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:379\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).Start.func1\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:265"}



2024-09-02T11:38:00.114743951Z {"level":"error","ts":"2024-09-02 11:38:00.114","logger":"collector|vsphere","msg":"full queue, event discarded","provider":"vms-poc/vspehere-hqidlinfvs51","error":"full queue, event discarded","stacktrace":"github.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*ClusterEventHandler).Error\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:385\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).notify\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:213\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Journal).Report\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:355\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).report\n\t/remote-source/app/pkg/lib/inventory/model/client.go:681\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit.func1\n\t/remote-source/app/pkg/lib/inventory/model/client.go:595\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Tx).Commit\n\t/remote-source/app/pkg/lib/inventory/model/client.go:611\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).validate\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:457\ngithub.com/konveyor/forklift-controller/pkg/controller/provider/container/vsphere.(*HostEventHandler).Updated\n\t/remote-source/app/pkg/controller/provider/container/vsphere/watch.go:417\ngithub.com/konveyor/forklift-controller/pkg/lib/inventory/model.(*Watch).Start.func1\n\t/remote-source/app/pkg/lib/inventory/model/journal.go:265"}

We notice that the Main container pod inside forklift is is getting runtime error: invalid memory address or nil pointer dereference

2024-09-02T11:34:46.531339970Z {"level":"info","ts":"2024-09-02 11:34:46.531","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"plan","object":{"name":"test-migration-vm-9-migration-2","namespace":"openshift-mtv"},"namespace":"openshift-mtv","name":"test-migration-vm-9-migration-2","reconcileID":"bd4459a0-89c2-4752-9821-b8b0dbc8d85d"}
2024-09-02T11:34:46.533466002Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2024-09-02T11:34:46.533466002Z  panic: runtime error: invalid memory address or nil pointer dereference
2024-09-02T11:34:46.533466002Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d34da2]
2024-09-02T11:34:46.533473974Z 
2024-09-02T11:34:46.533473974Z goroutine 271 [running]:
2024-09-02T11:34:46.533482847Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2024-09-02T11:34:46.533482847Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
2024-09-02T11:34:46.533486660Z panic({0x2c39780?, 0x4fa1b70?})
2024-09-02T11:34:46.533490313Z  /usr/lib/golang/src/runtime/panic.go:914 +0x21f
2024-09-02T11:34:46.533504419Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).mapDisks(0xc004b82fa8, 0x2cfd4c0?, {{0xc000f99d40, 0xa}, {0xc0008babd0, 0x13}, {0x0, 0x0}, {0x0, 0x0}}, ...)
2024-09-02T11:34:46.533514447Z  /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:706 +0x542
2024-09-02T11:34:46.533517733Z github.com/konveyor/forklift-controller/pkg/controller/plan/adapter/vsphere.(*Builder).VirtualMachine(0xc004b82fa8, {{0xc000f99d40, 0xa}, {0xc0008babd0, 0x13}, {0x0, 0x0}, {0x0, 0x0}}, 0xc004931308, ...)
2024-09-02T11:34:46.533527717Z  /remote-source/app/pkg/controller/plan/adapter/vsphere/builder.go:534 +0x485
2024-09-02T11:34:46.533535430Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).virtualMachine(0xc000675f98, 0xc000fa0ea0)
2024-09-02T11:34:46.533538693Z  /remote-source/app/pkg/controller/plan/kubevirt.go:1275 +0xaba
2024-09-02T11:34:46.533542288Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*KubeVirt).EnsureVM(0xc000675f98, 0xc000fa0ea0)
2024-09-02T11:34:46.533542288Z  /remote-source/app/pkg/controller/plan/kubevirt.go:389 +0x39a
2024-09-02T11:34:46.533611820Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).execute(0xc000675f80, 0xc000fa0ea0)
2024-09-02T11:34:46.533611820Z  /remote-source/app/pkg/controller/plan/migration.go:825 +0x17ce5
2024-09-02T11:34:46.533615427Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Migration).Run(0xc000675f80)
2024-09-02T11:34:46.533615427Z  /remote-source/app/pkg/controller/plan/migration.go:195 +0x285
2024-09-02T11:34:46.533622255Z github.com/konveyor/forklift-controller/pkg/controller/plan.(*Reconciler).execute(0xc003dd2c60, 0xc000e45c00)
2024-09-02T11:34:46.533622255Z  /remote-source/app/pkg/controller/plan/controller.go:428 +0x98b
2024-09-02T11:34:46.533626261Z github.com/konveyor/forklift-controller/pkg/controller/plan.Reconciler.Reconcile({{{0x3697178, 0xc000177700}, {0x36b2f40, 0xc0008da7e0}, {0x36a9c78, 0xc00016b5e0}}}, {0x36c23a0?, 0xc000c4dce8?}, {{{0xc000363ce0, 0xd}, ...}})
2024-09-02T11:34:46.533639832Z  /remote-source/app/pkg/controller/plan/controller.go:257 +0x9de
2024-09-02T11:34:46.533643352Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x36a7f48?, {0x36a4838?, 0xc003dd2c30?}, {{{0xc000363ce0?, 0xb?}, {0xc00057cb20?, 0x0?}}})
2024-09-02T11:34:46.533653431Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
2024-09-02T11:34:46.533656821Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006b5cc0, {0x36a4870, 0xc00075b6d0}, {0x2e5f7a0?, 0xc000c04260?})
2024-09-02T11:34:46.533663573Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3cc
2024-09-02T11:34:46.533667045Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006b5cc0, {0x36a4870, 0xc00075b6d0})
2024-09-02T11:34:46.533670539Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
2024-09-02T11:34:46.533673897Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
2024-09-02T11:34:46.533673897Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
2024-09-02T11:34:46.533677381Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 187
2024-09-02T11:34:46.533677381Z  /remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565

The pod memory usage:

# oc adm top pods -n openshift-mtv
NAME                                                    CPU(cores)   MEMORY(bytes)
forklift-api-85487cf4bc-sppcm                           0m           33Mi
forklift-controller-b7f544cdf-jm46p                     925m         927Mi  <=== 
forklift-operator-778b6b9d87-6ljkw                      0m           129Mi
forklift-ui-plugin-6856fc86fc-zbnkr                     0m           2Mi
forklift-validation-5fcb4c7bc5-jqqk9                    221m         119Mi
forklift-volume-populator-controller-66dbd8dffc-xj85l   1m           116Mi

Version:

Migration Tool Kit for Virtualization: "version":"2.6.4"

duplicates

MTV-1343 forklift-controller OOM during 100 VM/20+ plan migration from VMware

New

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates