Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28965

[IBMCloud] Cluster install failed and machine-api-controllers stucks in CrashLoopBackOff

XMLWordPrintable

    • Critical
    • No
    • CLOUD Sprint 249
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Machine API {ibm-cloud-title} controller did not integrate the full logging options from the klogr package.
      As a result, the controller crashed in Kubernetes version 1.29 and later.
      With this release, the missing options are included and the issue is resolved.
      (link:https://issues.redhat.com/browse/OCPBUGS-28965[*OCPBUGS-28965*])
      Show
      * Previously, the Machine API {ibm-cloud-title} controller did not integrate the full logging options from the klogr package. As a result, the controller crashed in Kubernetes version 1.29 and later. With this release, the missing options are included and the issue is resolved. (link: https://issues.redhat.com/browse/OCPBUGS-28965 [* OCPBUGS-28965 *])
    • Bug Fix
    • Done

      Description of problem:

      Cluster install failed on ibm cloud and machine-api-controllers stucks in CrashLoopBackOff

      Version-Release number of selected component (if applicable):

      from 4.16.0-0.nightly-2024-02-02-224339

      How reproducible:

      Always

      Steps to Reproduce:

          1. Install cluster on IBMCloud
    2.
    3.

      Actual results:

      Cluster install failed
$ oc get node                       
NAME                     STATUS   ROLES                  AGE     VERSION
maxu-16-gp2vp-master-0   Ready    control-plane,master   7h11m   v1.29.1+2f773e8
maxu-16-gp2vp-master-1   Ready    control-plane,master   7h11m   v1.29.1+2f773e8
maxu-16-gp2vp-master-2   Ready    control-plane,master   7h11m   v1.29.1+2f773e8

$ oc get machine -n openshift-machine-api           
NAME                           PHASE   TYPE   REGION   ZONE   AGE
maxu-16-gp2vp-master-0                                        7h15m
maxu-16-gp2vp-master-1                                        7h15m
maxu-16-gp2vp-master-2                                        7h15m
maxu-16-gp2vp-worker-1-xfvqq                                  7h5m
maxu-16-gp2vp-worker-2-5hn7c                                  7h5m
maxu-16-gp2vp-worker-3-z74z2                                  7h5m

openshift-machine-api                              machine-api-controllers-6cb7fcdcdb-k6sv2                     6/7     CrashLoopBackOff   92 (31s ago)     7h1m

$ oc logs -n openshift-machine-api  -c  machine-controller  machine-api-controllers-6cb7fcdcdb-k6sv2                          
I0204 10:53:34.336338       1 main.go:120] Watching machine-api objects only in namespace "openshift-machine-api" for reconciliation.panic: runtime error: invalid memory address or nil pointer dereference[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x285fe72]
goroutine 25 [running]:k8s.io/klog/v2/textlogger.(*tlogger).Enabled(0x0?, 0x0?)	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/k8s.io/klog/v2/textlogger/textlogger.go:81 +0x12sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc000438100, 0x0?)	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/sigs.k8s.io/controller-runtime/pkg/log/deleg.go:114 +0x92github.com/go-logr/logr.Logger.Info({{0x3232210?, 0xc000438100?}, 0x0?}, {0x2ec78f3, 0x17}, {0x0, 0x0, 0x0})	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/github.com/go-logr/logr/logr.go:276 +0x72sigs.k8s.io/controller-runtime/pkg/metrics/server.(*defaultServer).Start(0xc0003bd2c0, {0x322e350?, 0xc00058a140})	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/sigs.k8s.io/controller-runtime/pkg/metrics/server/server.go:185 +0x75sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1(0xc0002c4540)	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:223 +0xc8created by sigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile in goroutine 24	/go/src/github.com/openshift/machine-api-provider-ibmcloud/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:207 +0x19d

      Expected results:

       Cluster install succeed

      Additional info:

      may relate to this pr https://github.com/openshift/machine-api-provider-ibmcloud/pull/34

              mimccune@redhat.com Michael McCune
              rhn-support-zhsun Zhaohua Sun
              Zhaohua Sun Zhaohua Sun
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: