Uploaded image for project: 'OpenShift Hive'
  1. OpenShift Hive
  2. HIVE-2627

The hive-controllers crashing with invalid memory address

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • openshift-4.16
    • None
    • False
    • None
    • False
    • Important
    • Customer Facing, Customer Reported

      The hive-controllers crashing with invalid memory address

       

      After OCP upgrade to 4.16.10 the hive-controllers are in Crashloop. Logs shows following error:

       

      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5ce91d6]
      
      goroutine 1032 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
      panic({0x65f4900?, 0xaa11f90?})
              runtime/panic.go:770 +0x132
      github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain({{{{0xc007d7a7e0, 0x1a}, {0xc007d7a780, 0x1c}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...})
              github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:166 +0x56
      github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x70d0fbb, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...})
              github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x4e5
      github.com/openshift/hive/pkg/controller/machinepool.matchFailureDomains(_, {{{0xc00c55f3d0, 0xa}, {0xc007ca3120, 0x1c}}, {{0xc00c602588, 0x12}, {0x0, 0x0}, {0xc00c6025a0, ...}, ...}, ...}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:582 +0x294
      github.com/openshift/hive/pkg/controller/machinepool.matchMachineSets(_, {{{0xc00c55f3d0, 0xa}, {0xc007ca3120, 0x1c}}, {{0xc00c602588, 0x12}, {0x0, 0x0}, {0xc00c6025a0, ...}, ...}, ...}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:623 +0x145
      github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).syncMachineSets(0xc00888d8f0?, 0xc00d0ec200, 0xc000c72608, {0xc002883b88, 0x1, 0x0?}, 0xc00b878850, 0xc008b88488, {0x7db93c0, 0xc00c47e870}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:659 +0x219f
      github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).Reconcile(0xc0083fb220, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520?, 0x5?}, {0xc008f50510?, 0xc?}}})
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:343 +0x1b6e
      github.com/openshift/hive/pkg/controller/utils.delayingReconciler.Reconcile({{0x7d55b60?, 0xc0083fb220?}, {0x7dcbaa0?, 0xc0083fd030?}}, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520, 0x5}, {0xc008f50510, 0xc}}})
              github.com/openshift/hive/pkg/controller/utils/delayingreconciler.go:36 +0x86
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x7da4390?, {0x7d9a970?, 0xc007202300?}, {{{0xc008f50520?, 0xb?}, {0xc008f50510?, 0x0?}}})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}, {0x6998ee0, 0xc009519740})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 393
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x5ce91d6]
      
      goroutine 1033 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:116 +0x1e5
      panic({0x65f4900?, 0xaa11f90?})
              runtime/panic.go:770 +0x132
      github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.VSphereProviderConfig.ExtractFailureDomain({{{{0xc007d0cc60, 0x1a}, {0xc007d0cc20, 0x1c}}, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, ...}, ...}, ...})
              github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/vsphere.go:166 +0x56
      github.com/openshift/cluster-control-plane-machine-set-operator/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig.providerConfig.ExtractFailureDomain({{0x70d0fbb, 0x7}, {{{{...}, {...}}, {{...}, {...}, {...}, {...}, {...}, {...}, ...}, ...}}, ...})
              github.com/openshift/cluster-control-plane-machine-set-operator@v0.0.0-20240311123436-a58712e5c221/pkg/machineproviders/providers/openshift/machine/v1beta1/providerconfig/providerconfig.go:212 +0x4e5
      github.com/openshift/hive/pkg/controller/machinepool.matchFailureDomains(_, {{{0xc00c5843f0, 0xa}, {0xc007c38f40, 0x1c}}, {{0xc00c344fc0, 0x14}, {0x0, 0x0}, {0xc00c344fd8, ...}, ...}, ...}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:582 +0x294
      github.com/openshift/hive/pkg/controller/machinepool.matchMachineSets(_, {{{0xc00c5843f0, 0xa}, {0xc007c38f40, 0x1c}}, {{0xc00c344fc0, 0x14}, {0x0, 0x0}, {0xc00c344fd8, ...}, ...}, ...}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:623 +0x145
      github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).syncMachineSets(0xc008aab110?, 0xc009cb2a00, 0xc000f4ef08, {0xc00511d0b8, 0x1, 0x656e6e6f632b1a31?}, 0xc00c4a6000, 0xc00ccb6b48, {0x7db93c0, 0xc00bfe93b0}, ...)
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:659 +0x219f
      github.com/openshift/hive/pkg/controller/machinepool.(*ReconcileMachinePool).Reconcile(0xc0083fb220, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780?, 0x6?}, {0xc008f50770?, 0xe?}}})
              github.com/openshift/hive/pkg/controller/machinepool/machinepool_controller.go:343 +0x1b6e
      github.com/openshift/hive/pkg/controller/utils.delayingReconciler.Reconcile({{0x7d55b60?, 0xc0083fb220?}, {0x7dcbaa0?, 0xc0083fd030?}}, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780, 0x6}, {0xc008f50770, 0xe}}})
              github.com/openshift/hive/pkg/controller/utils/delayingreconciler.go:36 +0x86
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x7da4390?, {0x7d9a970?, 0xc0071956b0?}, {{{0xc008f50780?, 0xb?}, {0xc008f50770?, 0x0?}}})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:119 +0xb7
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0}, {0x6998ee0, 0xc0095197a0})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:316 +0x3bc
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00842f5e0, {0x7d9a9a8, 0xc0006a54f0})
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266 +0x1be
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227 +0x79
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 393
              sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:223 +0x50c

       

       

              efried.openshift Eric Fried
              rhn-support-mheppler Michal Heppler
              Jianping Shu Jianping Shu
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: