Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-3108

[KI] DS 3.1.0 workspace stuck in terminating state in Dashboard UI because of PVC problem

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Release Notes
    • Hide
      = Workspace starting and deletion might fail

      Currently, a workspace might fail to start, and then you might be unable to delete it. This is caused by a persistent volume claim (PVC) issue. Subsequently, the same problem occurs with other workspaces.

      .Workaround

      . Remove the `DevWorkspace` object of the first stuck workspace.
      . Remove the PVC that is bound to the invalid volume.
      Show
      = Workspace starting and deletion might fail Currently, a workspace might fail to start, and then you might be unable to delete it. This is caused by a persistent volume claim (PVC) issue. Subsequently, the same problem occurs with other workspaces. .Workaround . Remove the `DevWorkspace` object of the first stuck workspace. . Remove the PVC that is bound to the invalid volume.
    • Known Issue
    • Workaround Exists
    • Hide

      How to remove workspace completely:
      1) remove devworkspace object of stuck workspace
      2) remove PVC bound to invalid volume

      Show
      How to remove workspace completely: 1) remove devworkspace object of stuck workspace 2) remove PVC bound to invalid volume
    • Done
    • Hide

      1. Install DS 3.1.0.RC-07-25 on Single-Node OCP 4.10 deployed to PSI OSP by flexy-install script.
      2. Create and then delete workspace.

      It's still not clear what it's needed to do with "admin-devspaces" project to reproduce the error.

      Show
      1. Install DS 3.1.0.RC-07-25 on Single-Node OCP 4.10 deployed to PSI OSP by flexy-install script. 2. Create and then delete workspace. It's still not clear what it's needed to do with "admin-devspaces" project to reproduce the error.

    Description

      DS 3.1.0 workspace stack in terminating state in Dashboard UI:

      New workspace can't be started after that:

      There were "cleanup-workspace" pod in workspace namespace in “ContainerCreating” state, facing next errors:
      1) Invalid volume error

       {"code": 400, "message": "Invalid volume: volume 656eaec8-1b53-4d4d-9fff-2191f69b50e6 already attached"}}
      

      2) input/output error

      Error: relabel failed /var/lib/kubelet/pods/bae63ea6-cbff-427f-9bea-8f3cfec47d77/volumes/kubernetes.io~cinder/pvc-c0d90d43-47e1-4ae9-825b-440f854ede20: readdirent /var/lib/kubelet/pods/bae63ea6-cbff-427f-9bea-8f3cfec47d77/volumes/kubernetes.io~cinder/pvc-c0d90d43-47e1-4ae9-825b-440f854ede20: input/output error
      
      Stuck "java-gradle" devworkspace yaml
      apiVersion: workspace.devfile.io/v1alpha2
      kind: DevWorkspace
      metadata:
        annotations:
          che.eclipse.org/che-editor: eclipse/che-theia/latest
          che.eclipse.org/devfile-source: |
            factory:
              params: >-
                url=https://github.com/devspaces-samples/gs-validating-form-input/tree/devspaces-3.1-rhel-8&devWorkspace=https://devspaces.apps.ocp410-ltst-ds.crw-qe.com/devfile-registry/devfiles/03_java11-maven-gradle/devworkspace-che-theia-latest.yaml&storageType=persistent
          che.eclipse.org/last-updated-timestamp: '2022-08-02T07:22:30.470Z'
        deletionTimestamp: '2022-08-02T07:32:57Z'
        resourceVersion: '17040102'
        name: java-gradle
        uid: 64a0382e-6480-475a-b06f-9c78741c7238
        deletionGracePeriodSeconds: 0
        creationTimestamp: '2022-08-02T02:39:36Z'
        generation: 7
        managedFields:
          - apiVersion: workspace.devfile.io/v1alpha2
            fieldsType: FieldsV1
            fieldsV1:
              'f:metadata':
                'f:annotations':
                  .: {}
                  'f:che.eclipse.org/che-editor': {}
                  'f:che.eclipse.org/devfile-source': {}
                  'f:che.eclipse.org/last-updated-timestamp': {}
              'f:spec':
                .: {}
                'f:routingClass': {}
                'f:template':
                  .: {}
                  'f:commands': {}
                  'f:components': {}
                  'f:projects': {}
            manager: unknown
            operation: Update
            time: '2022-08-02T07:22:30Z'
          - apiVersion: workspace.devfile.io/v1alpha2
            fieldsType: FieldsV1
            fieldsV1:
              'f:status':
                'f:conditions': {}
                'f:message': {}
                'f:phase': {}
            manager: devworkspace-controller
            operation: Update
            subresource: status
            time: '2022-08-02T07:32:57Z'
        namespace: admin-devspaces
        finalizers:
          - storage.controller.devfile.io
        labels:
          controller.devfile.io/creator: caa1c31a-510d-45d9-9268-2afd15864a11
      spec:
        routingClass: che
        started: false
        template:
          commands:
            - exec:
                commandLine: gradle build
                component: tools
                group:
                  kind: build
                label: Build
                workingDir: '${PROJECTS_ROOT}/gs-validating-form-input/complete'
              id: 1-build
            - exec:
                commandLine: gradle bootRun
                component: tools
                group:
                  kind: run
                label: Run application
                workingDir: '${PROJECTS_ROOT}/gs-validating-form-input/complete'
              id: 2-run
            - exec:
                commandLine: gradle test
                component: tools
                group:
                  kind: run
                label: Run tests
                workingDir: '${PROJECTS_ROOT}/gs-validating-form-input/complete'
              id: 3-test
            - exec:
                commandLine: gradle bootRun --debug-jvm
                component: tools
                group:
                  kind: debug
                label: Run in debug mode
                workingDir: '${PROJECTS_ROOT}/gs-validating-form-input/complete'
              id: 4-debug
          components:
            - container:
                args:
                  - sh
                  - '-c'
                  - '${PLUGIN_REMOTE_ENDPOINT_EXECUTABLE}'
                endpoints:
                  - exposure: none
                    name: debug
                    protocol: tcp
                    targetPort: 5005
                  - exposure: public
                    name: input-form-endpoint
                    protocol: http
                    targetPort: 8080
                env:
                  - name: GRADLE_USER_HOME
                    value: /home/user/.gradle
                  - name: PLUGIN_REMOTE_ENDPOINT_EXECUTABLE
                    value: /remote-endpoint/plugin-remote-endpoint
                  - name: THEIA_PLUGINS
                    value: 'local-dir:///plugins/sidecars/tools'
                  - name: CHE_DASHBOARD_URL
                    value: 'https://devspaces.apps.ocp410-ltst-ds.crw-qe.com'
                  - name: CHE_PLUGIN_REGISTRY_URL
                    value: >-
                      https://devspaces.apps.ocp410-ltst-ds.crw-qe.com/plugin-registry/v3
                  - name: CHE_PLUGIN_REGISTRY_INTERNAL_URL
                    value: 'http://plugin-registry.openshift-devspaces.svc:8080/v3'
                image: >-
                  registry.redhat.io/devspaces/udi-rhel8@sha256:21a4e21e4a8e3a8ae721dfdbb80f74360a003a4aa57a729b4046839a28520a3b
                memoryLimit: 4Gi
                mountSources: true
                sourceMapping: /projects
                volumeMounts:
                  - name: gradle
                    path: /home/jboss/.gradle
                  - name: m2
                    path: /home/user/.m2
                  - name: remote-endpoint
                    path: /remote-endpoint
                  - name: plugins
                    path: /plugins
              name: tools
            - name: m2
              volume:
                size: 1G
            - name: gradle
              volume:
                size: 1G
            - name: theia-ide-java-gradle
              plugin:
                kubernetes:
                  name: theia-ide-java-gradle
          projects:
            - name: gs-validating-form-input
              zip:
                location: >-
                  http://devfile-registry.openshift-devspaces.svc:8080/resources/v2/gs-validating-form-input.zip
      status:
        conditions:
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: DevWorkspaceWarning
          - lastTransitionTime: '2022-08-02T07:32:57Z'
            status: Unknown
            type: FailedStart
          - lastTransitionTime: '2022-08-02T07:33:01Z'
            message: >-
              DevWorkspace common PVC cleanup job failed: see logs for job
              "cleanup-workspace64a0382e6480475a" for details. Additional information:
              Container cleanup-workspace64a0382e6480475a has state
              CreateContainerError
            status: 'True'
            type: Error
          - lastTransitionTime: '2022-08-02T07:32:57Z'
            message: Cleaning up resources for deletion
            status: 'True'
            type: Started
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: DevWorkspaceResolved
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: StorageReady
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: RoutingReady
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: ServiceAccountReady
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: PullSecretsReady
          - lastTransitionTime: '2022-08-02T07:32:31Z'
            status: Unknown
            type: DeploymentReady
        devworkspaceId: workspace64a0382e6480475a
        mainUrl: >-
          https://devspaces.apps.ocp410-ltst-ds.crw-qe.com/workspace64a0382e6480475a/theia-ide/3100/
        message: >-
          DevWorkspace common PVC cleanup job failed: see logs for job
          "cleanup-workspace64a0382e6480475a" for details. Additional information:
          Container cleanup-workspace64a0382e6480475a has state CreateContainerError
        phase: Error
      

      Attachments

        Activity

          People

            dnochevn Dmytro Nochevnov
            dnochevn Dmytro Nochevnov
            Max Leonov Max Leonov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: