Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-763

Partially-deployed CRW Operator will crash if checluster custom resource deleted

    XMLWordPrintable

Details

    • Hide
      • Install operator from OperatorHub
      • Create checluster (without proxy settings!)
      • Observe, that operator is stuck on Creating a new object Route, name: test (this is expected, as we "forget" to put proxy settings in Checluster CR)
      • Realize the mistake with proxy settings -> delete the checluster CR
      • Wait few moments and che-operator crashes.
      Show
      Install operator from OperatorHub Create checluster (without proxy settings!) Observe, that operator is stuck on Creating a new object Route, name: test (this is expected, as we "forget" to put proxy settings in Checluster CR) Realize the mistake with proxy settings -> delete the checluster CR Wait few moments and che-operator crashes.

    Description

      I've just experienced Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

      Whole operator log:

      time="2020-03-30T14:24:42Z" level=info msg="Default 'info' log level is applied" 
      time="2020-03-30T14:24:42Z" level=info msg="Go Version: go1.12.12" 
      time="2020-03-30T14:24:42Z" level=info msg="Go OS/Arch: linux/amd64" 
      time="2020-03-30T14:24:42Z" level=info msg="operator-sdk Version: v0.5.0" 
      time="2020-03-30T14:24:42Z" level=info msg="Operator is running on OpenShift v4.x" 
      time="2020-03-30T14:24:43Z" level=info msg="Registering Che Components Types" 
      time="2020-03-30T14:24:43Z" level=info msg="Starting the Cmd" 
      time="2020-03-30T14:25:59Z" level=info msg="Creating a test route test to extract router crt" 
      time="2020-03-30T14:25:59Z" level=info msg="Creating a new object Route, name: test" 
      time="2020-03-30T14:30:20Z" level=error msg="An error occurred when reaching test TLS route: Get https://test-crw.apps.rhopp-airgap2.qe.devcluster.openshift.com: dial tcp 18.189.114.225:443: connect: connection timed out" 
      time="2020-03-30T14:30:20Z" level=error msg="Failed to extract crt for secret self-signed-certificate. Failed to create a secret with a self signed crt: Get https://test-crw.apps.rhopp-airgap2.qe.devcluster.openshift.com: dial tcp 18.189.114.225:443: connect: connection timed out" 
      time="2020-03-30T14:30:21Z" level=info msg="Custom resource codeready-workspaces is being deleted. Deleting oAuthClient  first" 
      time="2020-03-30T14:30:21Z" level=error msg="Failed to Get oAuthClient : OAuthClient.oauth.openshift.io "" not found" 
      time="2020-03-30T14:30:21Z" level=info msg="Updating codeready-workspaces CR" 
      time="2020-03-30T14:30:21Z" level=info msg="Creating a test route test to extract router crt" 
      time="2020-03-30T14:30:22Z" level=error msg="Failed to get test route: Route.route.openshift.io "test" not found" 
      E0330 14:30:22.517476       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
      /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:522
      /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:82
      /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/signal_unix.go:390
      /go/src/github.com/eclipse/che-operator/pkg/controller/che/k8s_helpers.go:277
      /go/src/github.com/eclipse/che-operator/pkg/controller/che/create.go:338
      /go/src/github.com/eclipse/che-operator/pkg/controller/che/che_controller.go:326
      /go/src/github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213
      /go/src/github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
      /go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
      /opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/asm_amd64.s:1337
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
      	panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x11d6432]
      
      goroutine 834 [running]:
      github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
      	/go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x105
      panic(0x1336ca0, 0x233e5b0)
      	/opt/rh/go-toolset-1.12/root/usr/lib/go-toolset-1.12-golang/src/runtime/panic.go:522 +0x1b5
      github.com/eclipse/che-operator/pkg/controller/che.(*ReconcileChe).GetEndpointTlsCrt(0xc00069ab10, 0xc000565180, 0x0, 0x0, 0x3, 0x14c268d, 0x17, 0x16aeb20, 0xc000300c80)
      	/go/src/github.com/eclipse/che-operator/pkg/controller/che/k8s_helpers.go:277 +0xc62
      github.com/eclipse/che-operator/pkg/controller/che.(*ReconcileChe).CreateTLSSecret(0xc00069ab10, 0xc000565180, 0x0, 0x0, 0x14c268d, 0x17, 0xf, 0x16ae260)
      	/go/src/github.com/eclipse/che-operator/pkg/controller/che/create.go:338 +0x127
      github.com/eclipse/che-operator/pkg/controller/che.(*ReconcileChe).Reconcile(0xc00069ab10, 0xc0007396e9, 0x3, 0xc0007ecd40, 0x14, 0x2352ee0, 0x54, 0xc0000a4538, 0x0)
      	/go/src/github.com/eclipse/che-operator/pkg/controller/che/che_controller.go:326 +0x9d2
      github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005fc280, 0x0)
      	/go/src/github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:213 +0x213
      github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1()
      	/go/src/github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158 +0x36
      github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000360010)
      	/go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
      github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000360010, 0x3b9aca00, 0x0, 0x1, 0xc0000a44e0)
      	/go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xf8
      github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000360010, 0x3b9aca00, 0xc0000a44e0)
      	/go/src/github.com/eclipse/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
      created by github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
      	/go/src/github.com/eclipse/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:157 +0x311
      

      Environment:

      Disconnected OCP with proxy configured (spawned by Flexy job)

      Operator installed & mirrored according to https://docs.openshift.com/container-platform/4.3/operators/olm-restricted-networks.html

      For OLM application I've used https://quay.io/application/crw/codeready-workspaces (9.9.1585366605)

      See steps to reproduce to see how to reproduce this error. I understand it's not very easy - if you find it necessary, feel free to contact me and I can help you with reproducing it.

      Attachments

        Activity

          People

            abazko Anatolii Bazko
            rhopp@redhat.com Radim Hopp
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: