-
Bug
-
Resolution: Done
-
Major
-
None
-
openshift-4.16
-
None
-
False
-
None
-
False
-
-
-
Important
-
Customer Facing, Customer Reported
The hive-controllers crashing with invalid memory address
After OCP upgrade to 4.16.10 the hive-controllers are in Crashloop. Logs shows following error:
panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5ce91d6] goroutine 1032 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5 panic({0x65f4900?, 0xaa11f90?}) runtime/panic.go:770 +0x132 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain({{{{0xc007d7a7e0, 0x1a}, {0xc007d7a780, 0x1c}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...}) github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:166 +0x56 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x70d0fbb, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...}) github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x4e5 github.com/openshift/hive/pkg/controller/machinepool.matchFailureDomains(_, {{{0xc00c55f3d0, 0xa}, {0xc007ca3120, 0x1c}}, {{0xc00c602588, 0x12}, {0x0, 0x0}, {0xc00c6025a0, ...}, ...}, ...}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:582 +0x294 github.com/openshift/hive/pkg/controller/machinepool.matchMachineSets(_, {{{0xc00c55f3d0, 0xa}, {0xc007ca3120, 0x1c}}, {{0xc00c602588, 0x12}, {0x0, 0x0}, {0xc00c6025a0, ...}, ...}, ...}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:623 +0x145 github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).syncMachineSets(0xc00888d8f0?, 0xc00d0ec200, 0xc000c72608, {0xc002883b88, 0x1, 0x0?}, 0xc00b878850, 0xc008b88488, {0x7db93c0, 0xc00c47e870}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:659 +0x219f github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).Reconcile(0xc0083fb220, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520?, 0x5?}, {0xc008f50510?, 0xc?}}}) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:343 +0x1b6e github.com/openshift/hive/pkg/controller/utils.delayingReconciler.Reconcile({{0x7d55b60?, 0xc0083fb220?}, {0x7dcbaa0?, 0xc0083fd030?}}, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520, 0x5}, {0xc008f50510, 0xc}}}) github.com/openshift/hive/pkg/controller/utils/delayingreconciler.go:36 +0x86 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x7da4390?, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520?, 0xb?}, {0xc008f50510?, 0x0?}}}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}, {0x6998ee0, 0xc009519740}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 393 sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5ce91d6] goroutine 1033 [running]: sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5 panic({0x65f4900?, 0xaa11f90?}) runtime/panic.go:770 +0x132 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain({{{{0xc007d0cc60, 0x1a}, {0xc007d0cc20, 0x1c}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...}) github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:166 +0x56 github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x70d0fbb, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...}) github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x4e5 github.com/openshift/hive/pkg/controller/machinepool.matchFailureDomains(_, {{{0xc00c5843f0, 0xa}, {0xc007c38f40, 0x1c}}, {{0xc00c344fc0, 0x14}, {0x0, 0x0}, {0xc00c344fd8, ...}, ...}, ...}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:582 +0x294 github.com/openshift/hive/pkg/controller/machinepool.matchMachineSets(_, {{{0xc00c5843f0, 0xa}, {0xc007c38f40, 0x1c}}, {{0xc00c344fc0, 0x14}, {0x0, 0x0}, {0xc00c344fd8, ...}, ...}, ...}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:623 +0x145 github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).syncMachineSets(0xc008aab110?, 0xc009cb2a00, 0xc000f4ef08, {0xc00511d0b8, 0x1, 0x656e6e6f632b1a31?}, 0xc00c4a6000, 0xc00ccb6b48, {0x7db93c0, 0xc00bfe93b0}, ...) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:659 +0x219f github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).Reconcile(0xc0083fb220, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780?, 0x6?}, {0xc008f50770?, 0xe?}}}) github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:343 +0x1b6e github.com/openshift/hive/pkg/controller/utils.delayingReconciler.Reconcile({{0x7d55b60?, 0xc0083fb220?}, {0x7dcbaa0?, 0xc0083fd030?}}, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780, 0x6}, {0xc008f50770, 0xe}}}) github.com/openshift/hive/pkg/controller/utils/delayingreconciler.go:36 +0x86 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x7da4390?, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780?, 0xb?}, {0xc008f50770?, 0x0?}}}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}, {0x6998ee0, 0xc0095197a0}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}) sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79 created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 393 sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c
- blocks
-
HIVE-2616 Upgrade to Kubernetes 1.31
- Closed
- is related to
-
OCPBUGS-31808 control-plane-machine-set operator pod stuck into crashloopbackoff state with the nil pointer dereference runtime error
- Closed
- links to