Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-60688

WMCO panics with nil pointer dereference when reconciling services

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.20
    • Windows Containers
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • None
    • None
    • None
    • None
    • WINC - Sprint 276
    • 1
    • In Progress
    • Bug Fix
    • Hide
      *Cause*: If an internally used ConfigMap needs to be deleted and re-created, a nil error is dereferenced when logging the event.
      *Consequence*: The WMCO pod panics and restarts.
      *Fix*: The error handling logic was reworked
      *Result*: Operator pod no longer panics
      Show
      *Cause*: If an internally used ConfigMap needs to be deleted and re-created, a nil error is dereferenced when logging the event. *Consequence*: The WMCO pod panics and restarts. *Fix*: The error handling logic was reworked *Result*: Operator pod no longer panics
    • None
    • None
    • None
    • None

      Description of problem:

      The WMCO pod enters a crash loop with a nil pointer dereference panic when it's triggered to reconcile a service configuration change. The code in configmap_controller.go does not gracefully handle the error from the API call, leading it to operate on a nil object, which causes a panic.

      Version-Release number of selected component (if applicable):

          10.20.0-de978d2

      How reproducible:

          100%

      Steps to Reproduce:

          1. On a cluster with WMCO installed, remove the RBAC permissions that allow WMCO to read ConfigMaps from the openshift-machine-config-operator namespace: 
      oc delete role windows-machine-config-operator -n openshift-machine-config-operator --ignore-not-found
      oc delete rolebinding windows-machine-config-operator -n openshift-machine-config-operator --ignore-not-found    
          2. Trigger a reconciliation by creating a simple MachineConfig targeting the worker pool:
      cat <<EOF | oc apply -f -
      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        name: trigger-wmco-panic-bug-report
        labels:
          machineconfiguration.openshift.io/role: worker
      spec:
        config:
          ignition:
            version: 3.2.0
      EOF
          3. 
          

      Actual results:

      The WMCO pod panics and enters a crash loop. The logs show a runtime error: invalid memory address or nil pointer dereference stack trace originating from configmap_controller.go.

      Expected results:

      The WMCO should not panic.

      Additional info:

      {"level":"error","ts":"2025-08-20T13:27:01Z","msg":"Observed a panic","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-services-10.20.0-de978d2","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshift-windows-machine-config-operator","name":"windows-services-10.20.0-de978d2","reconcileID":"b1504f53-3e99-400c-ab01-c28b2dd10e8a","panic":"runtime error: invalid memory address or nil pointer dereference","panicGoValue":"\"invalid memory address or nil pointer dereference\"","stacktrace":"goroutine 500 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x45f32b0, 0xc002845ec0}, {0x3c5abc0, 0x5fd80f0})\n\t/build/windows-machine-config-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:108 +0x112\npanic({0x3c5abc0?, 0x5fd80f0?})\n\t/usr/lib/golang/src/runtime/panic.go:791 +0x132\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileServices(0xc00098bd60, {0x45f32b0, 0xc002845ec0}, 0xc0028f90e0)\n\t/build/windows-machine-config-operator/controllers/configmap_controller.go:218 +0x104\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile(0xc00098bd60, {0x45f32b0, 0xc002845ec0}, {{{0xc00005c050, 0x29}, {0xc00051e6c0, 0x20}}})\n\t/build/windows-machine-config-operator/controllers/configmap_controller.go:185 +0xa72\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc002845e30?, {0x45f32b0?, 0xc002845ec0?}, {{{0xc00005c050?, 0x0?}, {0xc00051e6c0?, 0x0?}}})\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x4631920, {0x45f32e8, 0xc0003a6320}, {{{0xc00005c050, 0x29}, {0xc00051e6c0, 0x20}}}, 0x0)\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:334 +0x3ad\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x4631920, {0x45f32e8, 0xc0003a6320})\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:294 +0x21e\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:255 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 208\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:251 +0x6b8\n","stacktrace":"runtime.sigpanic\n\t/usr/lib/golang/src/runtime/signal_unix.go:917\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).reconcileServices\n\t/build/windows-machine-config-operator/controllers/configmap_controller.go:218\ngithub.com/openshift/windows-machine-config-operator/controllers.(*ConfigMapReconciler).Reconcile\n\t/build/windows-machine-config-operator/controllers/configmap_controller.go:185\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:334\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:255"}
      {"level":"error","ts":"2025-08-20T13:27:01Z","msg":"Reconciler error","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","ConfigMap":{"name":"windows-services-10.20.0-de978d2","namespace":"openshift-windows-machine-config-operator"},"namespace":"openshift-windows-machine-config-operator","name":"windows-services-10.20.0-de978d2","reconcileID":"b1504f53-3e99-400c-ab01-c28b2dd10e8a","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:347\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:294\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:255"}
      {"level":"info","ts":"2025-08-20T13:27:16Z","logger":"controllers.configmap","msg":"Created","ConfigMap":{"name":"windows-services-10.20.0-de978d2","namespace":"openshift-windows-machine-config-operator"}}
      > oc get role windows-machine-config-operator -n openshift-machine-config-operator -o yaml
      Error from server (NotFound): roles.rbac.authorization.k8s.io "windows-machine-config-operator" not found
      >       oc get role -n openshift-windows-machine-config-operator windows-machine-config-operator -o yaml
      Error from server (NotFound): roles.rbac.authorization.k8s.io "windows-machine-config-operator" not found
      > oc get role -n openshift-windows-machine-config-operator windows-machine-config-operator -o yaml
      Error from server (NotFound): roles.rbac.authorization.k8s.io "windows-machine-config-operator" not found

              rh-ee-ssoto Sebastian Soto
              rrasouli Aharon Rasouli
              None
              None
              Aharon Rasouli Aharon Rasouli
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: