Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23913

machine-api-controller stuck in CrashLoopBackOff

XMLWordPrintable

    • ?
    • Important
    • No
    • ShiftStack Sprint 245
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      Due to a programming error in machine-api-controller, deleting a machine that didn't have a root volume set could trigger an unrecoverable error. As a consequence, such machine could permanently stay in "deleting" state and the corresponding Nova instance wouldn't be cleared.
      With this patch, machines that don't have a root volume are correctly deleted.
      Show
      Due to a programming error in machine-api-controller, deleting a machine that didn't have a root volume set could trigger an unrecoverable error. As a consequence, such machine could permanently stay in "deleting" state and the corresponding Nova instance wouldn't be cleared. With this patch, machines that don't have a root volume are correctly deleted.
    • Bug Fix

      Description of problem: Panic on machine-controller

      2023-11-23T18:18:47.899851056Z I1123 18:18:47.899752       1 controller.go:115]  "msg"="Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference" "controller"="machine-controller" "name"="bogus-6121tjfqk-cpr4v" "namespace"="openshift-machine-api" "object"={"name":"bogus-6121tjfqk-cpr4v","namespace":"openshift-machine-api"} "reconcileID"="38050b3e-3313-4500-8955-59f6822fd650"
2023-11-23T18:18:47.901976792Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2023-11-23T18:18:47.901976792Z 	panic: runtime error: invalid memory address or nil pointer dereference
2023-11-23T18:18:47.901976792Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x27fcb31]
2023-11-23T18:18:47.902001202Z 
2023-11-23T18:18:47.902001202Z goroutine 261 [running]:
2023-11-23T18:18:47.902001202Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
2023-11-23T18:18:47.902001202Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1fa
2023-11-23T18:18:47.902013625Z panic({0x2ab4640, 0x4373ed0})
2023-11-23T18:18:47.902022923Z 	/usr/lib/golang/src/runtime/panic.go:884 +0x213
2023-11-23T18:18:47.902043867Z github.com/openshift/machine-api-provider-openstack/pkg/machine.extractRootVolumeFromProviderSpec(...)
2023-11-23T18:18:47.902043867Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/convert.go:211
2023-11-23T18:18:47.902053364Z github.com/openshift/machine-api-provider-openstack/pkg/machine.(*OpenstackClient).Delete(0xc0000bfab0, {0x3113ff0?, 0xc000605ec0?}, 0xc00065fd40)
2023-11-23T18:18:47.902062370Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/pkg/machine/actuator.go:335 +0x1b1
2023-11-23T18:18:47.902082577Z github.com/openshift/machine-api-operator/pkg/controller/machine.(*ReconcileMachine).Reconcile(0xc000304aa0, {0x3113ff0, 0xc000605ec0}, {{{0xc000d66a50?, 0x0?}, {0xc000d66a38?, 0xc00043cd48?}}})
2023-11-23T18:18:47.902117667Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/github.com/openshift/machine-api-operator/pkg/controller/machine/controller.go:216 +0x1dee
2023-11-23T18:18:47.902139450Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x31181b8?, {0x3113ff0?, 0xc000605ec0?}, {{{0xc000d66a50?, 0xb?}, {0xc000d66a38?, 0x0?}}})
2023-11-23T18:18:47.902166210Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xc8
2023-11-23T18:18:47.902186773Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005488c0, {0x3113f48, 0xc000350550}, {0x2b9b6a0?, 0xc000475760?})
2023-11-23T18:18:47.902196557Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3ca
2023-11-23T18:18:47.902205655Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005488c0, {0x3113f48, 0xc000350550})
2023-11-23T18:18:47.902214747Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1d9
2023-11-23T18:18:47.902223782Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
2023-11-23T18:18:47.902223782Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x85
2023-11-23T18:18:47.902233237Z created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
2023-11-23T18:18:47.902242150Z 	/go/src/sigs.k8s.io/cluster-api-provider-openstack/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x587

      The bogus machine bogus-6121tjfqk-cpr4v was created by openstack-test "[sig-installer][Suite:openshift/openstack] Bugfix bz_2073398: [Serial] MachineSet scale-in does not leak OpenStack ports" which was run before and passed.

      Version-Release number of selected component (if applicable):

      • Network_Type: OVNKubernetes
      • osp_puddle: RHOS-17.1-RHEL-9-20231102.n.1
      • ocp_puddle: 4.15.0-0.nightly-2023-11-20-205649

      How reproducible: Observed once.
      Additional info: must-gather provided on private comment

            pprinett@redhat.com Pierre Prinetti
            rlobillo Ramón Lobillo
            Itshak Brown Itshak Brown
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: