Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-2367

CRW from latest channel failed to install when multiple CRW operators are there

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • 2.13.0.GA
    • 2.12.0.GA
    • docs
    • None

      CRW 2.12.0.RC-10-11 with operator image https://quay.io/crw/crw-2-rhel8-operator@sha256:458802564043f3047e477045c9cf4e06ef6b9e41bd8de3a05f482250f326ea28 installation failed on operator container creation:
      https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/CodereadyWorkspaces/job/e2e/job/basic/job/install-crw-using-crwctl/1530/console
      codeready-operator-9b95f58f8-2gjh2.pod.log

      operator log fragment
      I1012 04:11:32.506528       1 request.go:655] Throttling request took 2.340457604s, request: GET:https://172.30.0.1:443/apis/quota.openshift.io/v1?timeout=32s
      {"level":"error","ts":1634011894.4886575,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"org.eclipse.che","reconcilerKind":"CheCluster","controller":"checluster","name":"codeready-workspaces","namespace":"crw-crwctl","error":"Operation cannot be fulfilled on checlusters.org.eclipse.che \"codeready-workspaces\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/che-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/che-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"} 
      crwctl logs
       + /mnt/hudson_workspace/workspace/CodereadyWorkspaces/e2e/basic/install-crw-using-crwctl/crwctl/bin/crwctl server:deploy --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/CodereadyWorkspaces/e2e/basic/install-crw-using-crwctl/custom-resource-patch.yaml --no-olm-suggested-namespace --catalog-source-name=crw-2-12-osbs-latest --catalog-source-namespace=openshift-marketplace --package-manifest-name=codeready-workspaces --olm-channel=latest --listr-renderer=verbose -n crw-crwctl
       › Installer type is set to: 'olm'
       [04:03:27] Verify Kubernetes API [started]
       › Current Kubernetes context: 'default/api-ocp48-aws-crw-qe-com:6443/admin'
       [04:03:27] Verify Kubernetes API...OK [title changed]
       [04:03:28] Verify Kubernetes API...OK (it's OpenShift) [title changed]
       [04:03:28] Verify Kubernetes API...OK (it's OpenShift) [completed]
       [04:03:28]   Looking for an already existing CodeReady Workspaces instance [started]
       [04:03:28] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl" [started]
       [04:03:28] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl"...it is not [title changed]
       [04:03:28] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl"...it is not [completed]
       [04:03:28]   Looking for an already existing CodeReady Workspaces instance [completed]
       [04:03:28]   Openshift preflight checklist [started]
       [04:03:28] Verify if oc is installed [started]
       [04:03:28] Verify if oc is installed...done. [title changed]
       [04:03:28] Verify if oc is installed...done. [completed]
       [04:03:28] Verify if openshift is running [started]
       [04:03:29] Verify if openshift is running...done. [title changed]
       [04:03:29] Verify if openshift is running...done. [completed]
       [04:03:29] Check OpenShift version [started]
       [04:03:30] Check OpenShift version: 4.x [title changed]
       [04:03:30] Check OpenShift version: 4.x [completed]
       [04:03:30] Check Kubernetes version [started]
       [04:03:31] Check Kubernetes version: Unknown. [title changed]
       [04:03:31] Check Kubernetes version: Unknown. [completed]
       [04:03:31]   Openshift preflight checklist [completed]
       [04:03:31] Following CodeReady Workspaces logs [started]
       [04:03:31] Start following Operator logs [started]
       [04:03:31] Start following Operator logs...done [title changed]
       [04:03:31] Start following Operator logs...done [completed]
       [04:03:31] Start following CodeReady Workspaces Server logs [started]
       [04:03:31] Start following CodeReady Workspaces Server logs...done [title changed]
       [04:03:31] Start following CodeReady Workspaces Server logs...done [completed]
       [04:03:31] Start following PostgreSQL logs [started]
       [04:03:31] Start following PostgreSQL logs...done [title changed]
       [04:03:31] Start following PostgreSQL logs...done [completed]
       [04:03:31] Start following Keycloak logs [started]
       [04:03:31] Start following Keycloak logs...done [title changed]
       [04:03:31] Start following Keycloak logs...done [completed]
       [04:03:31] Start following Plug-in Registry logs [started]
       [04:03:31] Start following Plug-in Registry logs...done [title changed]
       [04:03:31] Start following Plug-in Registry logs...done [completed]
       [04:03:31] Start following Devfile Registry logs [started]
       [04:03:31] Start following Devfile Registry logs...done [title changed]
       [04:03:31] Start following Devfile Registry logs...done [completed]
       [04:03:31] Start following CodeReady Workspaces Dashboard logs [started]
       [04:03:31] Start following CodeReady Workspaces Dashboard logs...done [title changed]
       [04:03:31] Start following CodeReady Workspaces Dashboard logs...done [completed]
       [04:03:31] Start following namespace events [started]
       [04:03:31] Start following namespace events...done [title changed]
       [04:03:31] Start following namespace events...done [completed]
       [04:03:31] Following CodeReady Workspaces logs [completed]
       [04:03:31]   Running Olm installaion CodeReady Workspaces [started]
       [04:03:31] Check if OLM is pre-installed on the platform [started]
       [04:03:31] Check if OLM is pre-installed on the platform...done. [title changed]
       [04:03:31] Check if OLM is pre-installed on the platform...done. [completed]
       [04:03:31] Create Namespace (crw-crwctl) [started]
       [04:03:32] Create Namespace (crw-crwctl)...Done. [title changed]
       [04:03:32] Create Namespace (crw-crwctl)...Done. [completed]
       [04:03:32] Create operator group [started]
       [04:03:33] Create operator group...created new one. [title changed]
       [04:03:33] Create operator group...created new one. [completed]
       [04:03:33] Configure context information [started]
       [04:03:33] Configure context information...done. [title changed]
       [04:03:33] Configure context information...done. [completed]
       [04:03:33] Create operator subscription [started]
       [04:03:33] Create operator subscription...created new one. [title changed]
       [04:03:33] Create operator subscription...created new one. [completed]
       [04:03:33] Wait while subscription is ready [started]
       [04:03:34] Wait while subscription is ready...done. [title changed]
       [04:03:34] Wait while subscription is ready...done. [completed]
       [04:03:34] Wait while operator installed [started]
       [04:03:37] Wait while operator installed...done. [title changed]
       [04:03:37] Wait while operator installed...done. [completed]
       [04:03:37] Prepare CodeReady Workspaces cluster CR [started]
       [04:03:38] Prepare CodeReady Workspaces cluster CR...Done. [title changed]
       [04:03:38] Prepare CodeReady Workspaces cluster CR...Done. [completed]
       [04:03:38] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl [started]
       [04:03:39] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl...done. [title changed]
       [04:03:39] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl...done. [completed]
       [04:03:39]   Running Olm installaion CodeReady Workspaces [completed]
       [04:03:39]   Post installation checklist [started]
       [04:03:39] PostgreSQL pod bootstrap [started]
       [04:03:39] Scheduling [started]
       [04:31:25] Scheduling [failed]
       [04:31:25]  Timeout: there are no pods in the namespace: crw-crwctl, selector: app=codeready,component=postgres. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
       [04:31:25] PostgreSQL pod bootstrap [failed]
       [04:31:25]  Timeout: there are no pods in the namespace: crw-crwctl, selector: app=codeready,component=postgres. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
       [04:31:25]   Post installation checklist [failed]
       [04:31:25]  Timeout: there are no pods in the namespace: crw-crwctl, selector: app=codeready,component=postgres. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
           Error: Command server:deploy failed. Error log: 
           /home/hudson/.cache/crwctl/error.log.
      

      OCP version: 4.8

      Operator: quay.io/repository/crw/crw-2-rhel8-operator:2.12-9
      Recent changes: https://github.com/eclipse-che/che-operator/blob/main/main.go#L229

      Update: there was CRW from tech-preview channel installed before.
      Uninstallation didn't help.

      Update 2: following advice from sleshche, CRW has successfully finished installation after removal of all installed CRW 2.12.0 instances from OCP 4.8 cluster:

      I1012 16:21:46.761797       1 request.go:655] Throttling request took 1.014377982s, request: GET:https://172.30.0.1:443/apis/k8s.cni.cncf.io/v1?timeout=32s
      {"level":"error","ts":1634055713.5063508,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"org.eclipse.che","reconcilerKind":"CheCluster","controller":"checluster","name":"codeready-workspaces","namespace":"crw-crwctl","error":"Operation cannot be fulfilled on checlusters.org.eclipse.che \"codeready-workspaces\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/che-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/che-operator/vendor/sigs.k8s...
      I1012 16:21:56.762026       1 request.go:655] Throttling request took 2.248773151s, request: GET:https://172.30.0.1:443/apis/network.operator.openshift.io/v1?timeout=32s
      time="2021-10-12T16:22:02Z" level=info msg="Custom resource spec codeready-workspaces updated with openShiftoAuth: true"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.Secret, name: self-signed-certificate"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.Secret, name: openshift-api-crt"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.ConfigMap, name: ca-certs-merged"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.ServiceAccount, name: che"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.ClusterRole, name: crw-crwctl-cheworkspaces-clusterrole"
      time="2021-10-12T16:22:04Z" level=info msg="Creating a new object: v1.ClusterRoleBinding, name: crw-crwctl-cheworkspaces-clusterrole"
      time="2021-10-12T16:22:04Z" level=info msg="Added finalizer: cheWorkspaces.clusterpermissions.finalizers.che.eclipse.org"
      

      Update 3: installation of CRW 2.11.0 failed as well when there is CRW 2.12.0 instance exists on the same cluster:

      I1014 08:41:27.649758       1 request.go:655] Throttling request took 1.59789489s, request: GET:https://172.30.0.1:443/apis/scheduling.k8s.io/v1beta1?timeout=32s
      {"level":"error","ts":1634200890.3639247,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"org.eclipse.che","reconcilerKind":"CheCluster","controller":"checluster","name":"codeready-workspaces","namespace":"crw-2-11-to-12","error":"Operation cannot be fulfilled on checlusters.org.eclipse.che \"codeready-workspaces\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/che-operator/vendor/github.com/go-logr/zapr/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/che-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/che-operator/vendor/sigs...
      

              ffloreth@redhat.com Fabrice Flore-Thébault
              dnochevn Dmytro Nochevnov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: