Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-3441

[KI] It's not possible to re-deploy DWO operator after execution of "dsc server:delete --delete-all" command

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Release Notes
    • Hide
      = Redeploying the DevWorkspace Operator might fail

      Currently, redeploying the DevWorkspace Operator fails after running the `dsc server:delete --delete-all` command.

      .Workaround

      . Delete the remaining DevWorkspace Operator resources:
      +
      [source]
      ----
      workspaces=$(oc get devworkspaces --all-namespaces | awk 'NR>1 {print $1}') \
        && for workspace in $workspaces ; do \
             oc patch devworkspaces/$workspace -p '{ "metadata": { "finalizers": null }}' --type merge || true; \
             oc delete devworkspaces/$workspace || true; \
           done \
        && oc delete crd/devworkspaces.workspace.devfile.io
      ----
      +
      NOTE: If the `oc delete crd/devworkspaces.workspace.devfile.io` command fails, you must deep clean your OpenShift cluster.

      . Redeploy the DevWorkspace Operator.
      Show
      = Redeploying the DevWorkspace Operator might fail Currently, redeploying the DevWorkspace Operator fails after running the `dsc server:delete --delete-all` command. .Workaround . Delete the remaining DevWorkspace Operator resources: + [source] ---- workspaces=$(oc get devworkspaces --all-namespaces | awk 'NR>1 {print $1}') \   && for workspace in $workspaces ; do \        oc patch devworkspaces/$workspace -p '{ "metadata": { "finalizers": null }}' --type merge || true; \        oc delete devworkspaces/$workspace || true; \      done \   && oc delete crd/devworkspaces.workspace.devfile.io ---- + NOTE: If the `oc delete crd/devworkspaces.workspace.devfile.io` command fails, you must deep clean your OpenShift cluster. . Redeploy the DevWorkspace Operator.
    • Known Issue
    • Done
    • Hide

      1. Delete remaining DevWorkspace Operator resources:

      workspaces=$(oc get devworkspaces --all-namespaces | awk 'NR>1 {print $1}') \
  && for workspace in $workspaces ; do \
       oc patch devworkspaces/$workspace -p '{ "metadata": { "finalizers": null }}' --type merge || true; \
       oc delete devworkspaces/$workspace || true; \
     done \
  && oc delete crd/devworkspaces.workspace.devfile.io

      2. Redeploy DevWorkspace Operator.

      Note: if "oc delete crd/devworkspaces.workspace.devfile.io" failed, you need deep clean up of cluster.

      Show
      1. Delete remaining DevWorkspace Operator resources: workspaces=$(oc get devworkspaces --all-namespaces | awk 'NR>1 {print $1}' ) \ && for workspace in $workspaces ; do \ oc patch devworkspaces/$workspace -p '{ "metadata" : { "finalizers" : null }}' --type merge || true ; \ oc delete devworkspaces/$workspace || true ; \ done \ && oc delete crd/devworkspaces.workspace.devfile.io 2. Redeploy DevWorkspace Operator. Note : if "oc delete crd/devworkspaces.workspace.devfile.io" failed, you need deep clean up of cluster.

    Description

      Test scenario:
      1. Open OCP 4.11
      2. Install DWO 0.16.0.RC-10-14
      3. Install DS 3.2.0.RC-10-12_163
      4. Create new workspaces.
      5. Delete DS 3.2.0 using "dsc server:delete --delete-all" command (dsc/3.2.0-CI-ce41-redhat.a05925f linux-x64 node-v16.13.2)
      6. Try to install DWO 0.16.0.RC-10-14 again.

      Observed result:
      pending DWO installation

      error validating existing CRs against new CRD's schema for "devworkspaces.workspace.devfile.io": error listing resources in GroupVersionResource schema.GroupVersionResource

      Unknown macro: {Group}

      : conversion webhook for workspace.devfile.io/v1alpha2, Kind=DevWorkspace failed: Post "https://devworkspace-controller-manager-service.openshift-operators.svc:443/convert?timeout=30s": no endpoints available for service "devworkspace-controller-manager-service"

      clusterserviceversion-devworkspace-operator.v0.16.0.yaml

      dsc server:delete command logs
       + /mnt/hudson_workspace/workspace/Testing/e2e/basic/install-ds/dsc/bin/dsc server:delete --delete-all --yes --listr-renderer=verbose
 [10:45:10] Verify Kubernetes API [started]
 › Current Kubernetes context: 'default/api-ocp411-aws-crw-qe-com:6443/admin'
 [10:45:10] Verify Kubernetes API...[OK] [title changed]
 [10:45:10] Verify Kubernetes API...[OK] [completed]
 [10:45:10] Uninstall Dev Workspace Operator [started]
 [10:45:11] Delete Custom Resources [started]
 [10:45:11] Delete workspace.devfile.io/v1alpha1 resources [started]
 [10:45:11] Delete workspace.devfile.io/v1alpha1 resources...[Failed: conversion webhook for workspace.devfile.io/v1alpha2, Kind=DevWorkspace failed: Post "https://devworkspace-controller-manager-service.openshift-operators.svc:443/convert?timeout=30s": service "devworkspace-controller-manager-service" not found] [title changed]
 [10:45:11] Delete workspace.devfile.io/v1alpha1 resources...[Failed: conversion webhook for workspace.devfile.io/v1alpha2, Kind=DevWorkspace failed: Post "https://devworkspace-controller-manager-service.openshift-operators.svc:443/convert?timeout=30s": service "devworkspace-controller-manager-service" not found] [completed]
 [10:45:11] Delete controller.devfile.io/v1alpha1 resources [started]
 [10:45:24] Delete controller.devfile.io/v1alpha1 resources...[Ok] [title changed]
 [10:45:24] Delete controller.devfile.io/v1alpha1 resources...[Ok] [completed]
 [10:45:24] Delete Custom Resources [completed]
 [10:45:24] Delete operator resources [started]
 [10:45:24] Delete WebhookConfigurations controller.devfile.io [started]
 [10:45:25] Delete WebhookConfigurations controller.devfile.io ...[Ok] [title changed]
 [10:45:25] Delete WebhookConfigurations controller.devfile.io ...[Ok] [completed]
 [10:45:25] Delete CRDs [started]
 [10:45:26] Delete CRDs...[Ok] [title changed]
 [10:45:26] Delete CRDs...[Ok] [completed]
 [10:45:26] Delete Deployments [started]
 [10:45:27] Delete Deployments...[Ok] [title changed]
 [10:45:27] Delete Deployments...[Ok] [completed]
 [10:45:27] Delete Services [started]
 [10:45:27] Delete Services...[Ok] [title changed]
 [10:45:27] Delete Services...[Ok] [completed]
 [10:45:27] Delete Secrets [started]
 [10:45:28] Delete Secrets...[Ok] [title changed]
 [10:45:28] Delete Secrets...[Ok] [completed]
 [10:45:28] Delete RoleBindings [started]
 [10:45:28] Delete RoleBindings...[Ok] [title changed]
 [10:45:28] Delete RoleBindings...[Ok] [completed]
 [10:45:28] Delete Roles [started]
 [10:45:29] Delete Roles...[Ok] [title changed]
 [10:45:29] Delete Roles...[Ok] [completed]
 [10:45:29] Delete ClusterRoleBindings [started]
 [10:45:29] Delete ClusterRoleBindings...[Ok] [title changed]
 [10:45:29] Delete ClusterRoleBindings...[Ok] [completed]
 [10:45:29] Delete ClusterRoles [started]
 [10:45:31] Delete ClusterRoles...[Ok] [title changed]
 [10:45:31] Delete ClusterRoles...[Ok] [completed]
 [10:45:31] Delete ServiceAccounts [started]
 [10:45:31] Delete ServiceAccounts...[Ok] [title changed]
 [10:45:31] Delete ServiceAccounts...[Ok] [completed]
 [10:45:31] Delete Issuer devworkspace-controller-selfsigned-issuer [started]
 [10:45:31] Delete Issuer devworkspace-controller-selfsigned-issuer...[Ok] [title changed]
 [10:45:31] Delete Issuer devworkspace-controller-selfsigned-issuer...[Ok] [completed]
 [10:45:31] Delete Certificate devworkspace-controller-serving-cert [started]
 [10:45:32] Delete Certificate devworkspace-controller-serving-cert...[Ok] [title changed]
 [10:45:32] Delete Certificate devworkspace-controller-serving-cert...[Ok] [completed]
 [10:45:32] Delete operator resources [completed]
 [10:45:32] Uninstall Dev Workspace Operator [completed]
 [10:45:32] Uninstall Red Hat OpenShift Dev Spaces Operator [started]
 [10:45:32] Delete Custom Resources [started]
 [10:45:32] Delete org.eclipse.che/v2 resources [started]
 [10:45:33] Delete org.eclipse.che/v2 resources...[Ok] [title changed]
 [10:45:33] Delete org.eclipse.che/v2 resources...[Ok] [completed]
 [10:45:33] Delete Custom Resources [completed]
 [10:45:33] Delete OLM resources [started]
 [10:45:33] Check if OLM is pre-installed on the platform [started]
 [10:45:33] Check if OLM is pre-installed on the platform...[Found] [title changed]
 [10:45:33] Check if OLM is pre-installed on the platform...[Found] [completed]
 [10:45:33] Delete Subscription [started]
 [10:45:33] Delete Subscription...[Ok] [title changed]
 [10:45:33] Delete Subscription...[Ok] [completed]
 [10:45:33] Delete ClusterServiceVersion [started]
 [10:45:34] Delete ClusterServiceVersion...[Ok] [title changed]
 [10:45:34] Delete ClusterServiceVersion...[Ok] [completed]
 [10:45:34] Delete CatalogSources [started]
 [10:45:34] Delete CatalogSources...[Ok] [title changed]
 [10:45:34] Delete CatalogSources...[Ok] [completed]
 [10:45:34] Delete Role prometheus-k8s [started]
 [10:45:35] Delete Role prometheus-k8s...[Ok] [title changed]
 [10:45:35] Delete Role prometheus-k8s...[Ok] [completed]
 [10:45:35] Delete RoleBinding prometheus-k8s [started]
 [10:45:35] Delete RoleBinding prometheus-k8s...[Ok] [title changed]
 [10:45:35] Delete RoleBinding prometheus-k8s...[Ok] [completed]
 [10:45:35] Delete OLM resources [completed]
 [10:45:35] Delete operator resources [started]
 [10:45:35] Delete ValidatingWebhookConfiguration org.eclipse.che [started]
 [10:45:35] Delete ValidatingWebhookConfiguration org.eclipse.che...[Ok] [title changed]
 [10:45:35] Delete ValidatingWebhookConfiguration org.eclipse.che...[Ok] [completed]
 [10:45:35] Delete CRDs [started]
 [10:45:35] Delete CRDs...[Ok] [title changed]
 [10:45:35] Delete CRDs...[Ok] [completed]
 [10:45:35] Delete Deployments [started]
 [10:45:35] Delete Deployments...[Ok] [title changed]
 [10:45:35] Delete Deployments...[Ok] [completed]
 [10:45:35] Delete Services [started]
 [10:45:36] Delete Services...[Ok] [title changed]
 [10:45:36] Delete Services...[Ok] [completed]
 [10:45:36] Delete Ingresses [started]
 [10:45:36] Delete Ingresses...[Ok] [title changed]
 [10:45:36] Delete Ingresses...[Ok] [completed]
 [10:45:36] Delete Secrets [started]
 [10:45:36] Delete Secrets...[Ok] [title changed]
 [10:45:36] Delete Secrets...[Ok] [completed]
 [10:45:36] Delete ConfigMaps [started]
 [10:45:37] Delete ConfigMaps...[Ok] [title changed]
 [10:45:37] Delete ConfigMaps...[Ok] [completed]
 [10:45:37] Delete RoleBindings [started]
 [10:45:41] Delete RoleBindings...[Ok] [title changed]
 [10:45:41] Delete RoleBindings...[Ok] [completed]
 [10:45:41] Delete Roles [started]
 [10:45:41] Delete Roles...[Ok] [title changed]
 [10:45:41] Delete Roles...[Ok] [completed]
 [10:45:41] Delete ClusterRoles [started]
 [10:45:43] Delete ClusterRoles...[Ok] [title changed]
 [10:45:43] Delete ClusterRoles...[Ok] [completed]
 [10:45:43] Delete ClusterRoleBindings [started]
 [10:45:44] Delete ClusterRoleBindings...[Ok] [title changed]
 [10:45:44] Delete ClusterRoleBindings...[Ok] [completed]
 [10:45:44] Delete ServiceAccounts [started]
 [10:45:45] Delete ServiceAccounts...[Ok] [title changed]
 [10:45:45] Delete ServiceAccounts...[Ok] [completed]
 [10:45:45] Delete Issuer devspaces-operator-selfsigned-issuer [started]
 [10:45:45] Delete Issuer devspaces-operator-selfsigned-issuer...[Ok] [title changed]
 [10:45:45] Delete Issuer devspaces-operator-selfsigned-issuer...[Ok] [completed]
 [10:45:45] Delete Certificate devspaces-operator-serving-cert [started]
 [10:45:45] Delete Certificate devspaces-operator-serving-cert...[Ok] [title changed]
 [10:45:45] Delete Certificate devspaces-operator-serving-cert...[Ok] [completed]
 [10:45:45] Delete PVCs [started]
 [10:45:45] Delete PVCs...[Ok] [title changed]
 [10:45:45] Delete PVCs...[Ok] [completed]
 [10:45:45] Delete ConsoleLink che [started]
 [10:45:46] Delete ConsoleLink che...[Ok] [title changed]
 [10:45:46] Delete ConsoleLink che...[Ok] [completed]
 [10:45:46] Delete operator resources [completed]
 [10:45:46] Wait until all pods are deleted [started]
 [10:45:46] Che Server pod [started]
 [10:45:46] Che Server pod...[Ok] [title changed]
 [10:45:46] Che Server pod...[Ok] [completed]
 [10:45:46] Dashboard pod [started]
 [10:45:46] Dashboard pod...[Ok] [title changed]
 [10:45:46] Dashboard pod...[Ok] [completed]
 [10:45:46] PostgreSQL pod [started]
 [10:45:46] PostgreSQL pod...[Ok] [title changed]
 [10:45:46] PostgreSQL pod...[Ok] [completed]
 [10:45:46] Devfile Registry pod [started]
 [10:45:46] Devfile Registry pod...[Ok] [title changed]
 [10:45:46] Devfile Registry pod...[Ok] [completed]
 [10:45:46] Plug-in Registry [started]
 [10:45:47] Plug-in Registry...[Ok] [title changed]
 [10:45:47] Plug-in Registry...[Ok] [completed]
 [10:45:47] Wait until all pods are deleted [completed]
 [10:45:47] Delete namespace openshift-devspaces [started]
 [10:45:47] Delete namespace openshift-devspaces [completed]
 [10:45:47] Uninstall Red Hat OpenShift Dev Spaces Operator [completed]
 Command server:delete has completed successfully in 00:37.

      Test pipeline: https://main-jenkins-csb-crwqe.apps.ocp-c1.prod.psi.redhat.com/job/Testing/job/e2e/job/basic/job/install-ds/1072/console

      Case 2:

      https://main-jenkins-csb-crwqe.apps.ocp-c1.prod.psi.redhat.com/job/Testing/job/e2e/job/basic/job/install-ds/1089/console

      Remained DWO resources:

      $ oc get devworkspaces -n admin-devspaces
NAME                    DEVWORKSPACE ID             PHASE     INFO
java-gradle             workspaced96226f01e8b4d4c   Failed    Container tools has state ImagePullBackOff
java-lombok             workspaceda2e2a739a7444b4   Running   https://devspaces.apps.ocp411.crw-qe.com/workspaceda2e2a739a7444b4/idea-rhel8/8887/?backgroundColor=434343&wss
jboss-eap-quickstarts   workspacebfe11afe979e4e8a   Failed    devworkspace failed to progress past phase 'Starting' for longer than timeout (10m)

$ oc get crd | grep devworkspace
devworkspaces.workspace.devfile.io                                2022-10-12T16:22:18Z

      Update: the issue hasn't been reproduced on OCP 4.10: https://main-jenkins-csb-crwqe.apps.ocp-c1.prod.psi.redhat.com/job/Testing/job/e2e/job/basic/job/install-ds/1078/console

      Attachments

        Issue Links

          is related to

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) CRW-3203 [RN] Remove workspaces and CRDs when removing dev spaces

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. CRW-3536 "dcs server:delete --delete-all" command didn't remove remaining "devworkspaceoperatorconfig" resource

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          mentioned on

          GitLab Merge request - Create 'admin-devspaces' namespace in api-tests pipeline; use 'dsc server:delete --delete-all' command

          GitLab Merge request - Don't use '--delete-all' parameter in test pipelines with special openshift-devspaces project configuration

          Activity

            People

              abazko Anatolii Bazko
              dnochevn Dmytro Nochevnov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: