Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-957

Can't start new workspace because mkdir pod failed to start because of PVC binding timeout

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Blocker Blocker
    • 2.2.0.GA
    • 2.2.0.GA
    • testing
    • None

      It is impossible to start any CRW 2.2.0.RC0-09.06 workspace on QE OCP 4.4 on CRW OpenStack:

      codeready server log
      2020-06-10 13:01:03,672[io-8080-exec-10]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 479]   - Starting workspace 'admin/cakephp-ex-vj473' with id 'workspace5qxy6w0p18yy81i0' by user 'admin'
      2020-06-10 13:06:05,217[aceSharedPool-0]  [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223]  - Unable to perform '[mkdir, -m, 777, -p, /tmp/job_mount/workspace5qxy6w0p18yy81i0/che-logs-che-plugin-broker/]' command for the workspace 'workspace5qxy6w0p18yy81i0' cause: 'Waiting for pod 'mkdir-workspace5qxy6w0p18yy81i0' reached timeout'
      
      mkdir logs
      3m20s       Warning   FailedMount              pod/mkdir-workspace5qxy6w0p18yy81i0                                   Unable to attach or mount volumes: unmounted volumes=[claim-che-workspace-workspace5qxy6w0p18yy81i0], unattached volumes=[claim-che-workspace-workspace5qxy6w0p18yy81i0 default-token-cbdxr]: timed out waiting for the condition
      
      events
      16m         Normal    WaitForFirstConsumer     persistentvolumeclaim/claim-che-workspace-workspace5qxy6w0p18yy81i0   waiting for first consumer to be created before binding
      16m         Normal    ProvisioningSucceeded    persistentvolumeclaim/claim-che-workspace-workspace5qxy6w0p18yy81i0   Successfully provisioned volume pvc-f33da6e3-475f-4306-858e-94dff0faff70 using kubernetes.io/cinder
      <unknown>   Normal    Scheduled                pod/mkdir-workspace5qxy6w0p18yy81i0                                   Successfully assigned admin-codeready/mkdir-workspace5qxy6w0p18yy81i0 to ocp44-2pxrq-worker-f288w
      8m45s       Warning   FailedMount              pod/mkdir-workspace5qxy6w0p18yy81i0                                   Unable to attach or mount volumes: unmounted volumes=[claim-che-workspace-workspace5qxy6w0p18yy81i0], unattached volumes=[default-token-cbdxr claim-che-workspace-workspace5qxy6w0p18yy81i0]: timed out waiting for the condition
      2m58s       Warning   FailedAttachVolume       pod/mkdir-workspace5qxy6w0p18yy81i0                                   AttachVolume.Attach failed for volume "pvc-f33da6e3-475f-4306-858e-94dff0faff70" : Volume "4534939c-ba21-44a9-bce0-356314b9c043" failed to be attached within the alloted time
      115s        Warning   FailedMount              pod/mkdir-workspace5qxy6w0p18yy81i0                                   Unable to attach or mount volumes: unmounted volumes=[claim-che-workspace-workspace5qxy6w0p18yy81i0], unattached volumes=[claim-che-workspace-workspace5qxy6w0p18yy81i0 default-token-cbdxr]: timed out waiting for the condition
      

      CRW 2.2.0.RC0-09.06 bits:

      • quay.io/crw/server-rhel8:2.2-8 - based on Eclipse Che 7.14.1
      • quay.io/repository/crw/crw-2-rhel8-operator:2.2-11

      PVC remained in Terminating state:

      There is the same StorageClass on QE OCP 4.4 instance for a last 2 weeks, and we didn't have a problem in CRW 2.2.0.RC0 based on 7.13.0:
      https://console-openshift-console.apps.ocp44.crw-qe.com/k8s/cluster/storageclasses/standard/yaml

      kind: StorageClass
      apiVersion: storage.k8s.io/v1
      metadata:
        name: standard
        selfLink: /apis/storage.k8s.io/v1/storageclasses/standard
        uid: dbe2ef90-07b9-454d-895c-9812c12da7a2
        resourceVersion: '10904'
        creationTimestamp: '2020-05-20T12:14:13Z'
        annotations:
          storageclass.kubernetes.io/is-default-class: 'true'
        ownerReferences:
          - apiVersion: v1
            kind: clusteroperator
            name: storage
            uid: 56778f8c-98e6-4662-b25f-9f86676fe789
      provisioner: kubernetes.io/cinder
      reclaimPolicy: Delete
      allowVolumeExpansion: true
      volumeBindingMode: WaitForFirstConsumer
      

      The issue hasn't been reproduced on QE OCP 4.5 deployed to AWS.

            rhopp@redhat.com Radim Hopp
            dnochevn Dmytro Nochevnov
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: