Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13790

Segmentation Violation found in WMCO .ensureWICDSecretContent

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 4.12.z
    • 4.12
    • Windows Containers
    • None
    • Critical
    • No
    • 3
    • WINC - Sprint 236, WINC - Sprint 237
    • 2
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      
      Running regression testing for WMCO 7.1.0 (windows-machine-config-operator-container-7.1.0-36 and windows-machine-config-operator-bundle-container-v7.1.0-35) it was found that none of the tests were passing in Azure with Windows Server 2019. When checking the WMCO status, it was in CrashLoopBackOff:
      $ oc get pods -n openshift-windows-machine-config-operator
      NAME                                              READY   STATUS             RESTARTS       AGE
      windows-machine-config-operator-f8dd9dd84-rc2kz   0/1     CrashLoopBackOff   10 (93s ago)   46m
      
      And the logs were displaying a Segmentation Violation log:
      
      Eventually, the container goes back into Running state but crashes again, ending in a never ending loop of restarts.
      
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting Controller","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest"}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting Controller","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine"}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting EventSource","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","source":"kind source: *v1.Node"}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting EventSource","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","source":"kind source: *v1.Node"}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting Controller","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap"}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting workers","controller":"certificatesigningrequest","controllerGroup":"certificates.k8s.io","controllerKind":"CertificateSigningRequest","worker count":1}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting workers","controller":"configmap","controllerGroup":"","controllerKind":"ConfigMap","worker count":1}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting workers","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","worker count":1}
      {"level":"info","ts":"2023-05-18T12:36:28Z","msg":"Starting workers","controller":"secret","controllerGroup":"","controllerKind":"Secret","worker count":1}
      {"level":"info","ts":"2023-05-18T12:36:43Z","logger":"controller.windowsmachine","msg":"processing","windowsmachine":"openshift-machine-api/windows-n69vt","address":"10.0.128.7"}
      {"level":"info","ts":"2023-05-18T12:37:18Z","logger":"wc 10.0.128.7","msg":"configuring"}
      {"level":"info","ts":"2023-05-18T12:37:34Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"machine","controllerGroup":"machine.openshift.io","controllerKind":"Machine","machine":{"name":"windows-n69vt","namespace":"openshift-machine-api"},"namespace":"openshift-machine-api","name":"windows-n69vt","reconcileID":"da85fc1b-e1c6-4221-8e46-843954ba1c8c"}
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x16b7d7a]
      
      goroutine 982 [running]:
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0x1f4
      panic({0x1ad6200, 0x2f40420})
              /usr/lib/golang/src/runtime/panic.go:884 +0x212
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).ensureWICDSecretContent(0x2089910?, 0x0)
              /remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:935 +0x5a
      github.com/openshift/windows-machine-config-operator/pkg/windows.(*windows).Bootstrap(0xc0025e4d80, {0x2066f28, 0xd}, {0xc000462190, 0x43}, {0xc00005c010, 0x29}, 0xc001ad6570?)
              /remote-source/build/windows-machine-config-operator/pkg/windows/windows.go:432 +0xdb
      github.com/openshift/windows-machine-config-operator/pkg/nodeconfig.(*nodeConfig).Configure(0xc000454780)
              /remote-source/build/windows-machine-config-operator/pkg/nodeconfig/nodeconfig.go:155 +0x350
      github.com/openshift/windows-machine-config-operator/controllers.(*instanceReconciler).ensureInstanceIsUpToDate(0xc0005141b0, 0xc002622640, 0x1dc9485?, 0x2a?)
              /remote-source/build/windows-machine-config-operator/controllers/controllers.go:84 +0x5f7
      github.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).configureMachine(0xc0005141b0, {0xc0025b6c40?, 0x1d8b5f2?}, {0xc002530a8d, 0xd}, {0xc0025b6bc0?, 0x2?}, 0x2?)
              /remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:432 +0x273
      github.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile(0xc0005141b0, {0x2086718, 0xc00213fe60}, {{{0xc0023a0a50, 0x15}, {0xc0027bb050, 0xd}}})
              /remote-source/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:354 +0xea9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2086670?, {0x2086718?, 0xc00213fe60?}, {{{0xc0023a0a50?, 0x1c9d380?}, {0xc0027bb050?, 0x4045d4?}}})
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121 +0xc8
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005a3d60, {0x2086670, 0xc000c811c0}, {0x1b75640?, 0xc0000a20e0?})
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320 +0x33c
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005a3d60, {0x2086670, 0xc000c811c0})
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 +0x1d9
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234 +0x85
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
              /remote-source/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:230 +0x333
      
      
      

      Version-Release number of selected component (if applicable):

      $ oc get cm -n openshift-windows-machine-config-operator 
      NAME                                   DATA   AGE
      kube-root-ca.crt                       1      52m
      openshift-service-ca.crt               1      52m
      windows-machine-config-operator-lock   0      52m
      windows-services-7.1.0-39ce397         2      52m
      [cloud-user@preserve-jfrancoa openshift-tests-private]$ oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.12.0-0.nightly-2023-05-17-133811   True        False         5h6m    Cluster version is 4.12.0-0.nightly-2023-05-17-133811
      
      

      How reproducible:

      Not sure
      

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

      WMCO crashes
      

      Expected results:

      WMCO runs correctly.
      

      Additional info:

      
      

              rh-ee-ssoto Sebastian Soto
              rhn-engineering-jfrancoa Jose Luis Franco Arza (Inactive)
              Aharon Rasouli Aharon Rasouli
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: