Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-4572

Dev Spaces 3.7.0 failed to deploy to test OCP instances using brew.registry.redhat.io mirroring

XMLWordPrintable

    • False
    • None
    • False

      There was Dev Spaces 3.7.0 deployment error:
      https://main-jenkins-csb-crwqe.apps.ocp-c1.prod.psi.redhat.com/job/Testing/job/e2e/job/basic/job/install-ds/2543/console

      dsc logs
       + /mnt/hudson_workspace/workspace/Testing/e2e/basic/install-ds/dsc/bin/dsc server:deploy --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/Testing/e2e/basic/install-ds/custom-resource-patch.yaml --package-manifest-name=devspaces --catalog-source-name=devspaces-operator-3-7-osbs-stable --olm-channel=stable --catalog-source-namespace=openshift-marketplace --listr-renderer=verbose
       [08:32:41] Verify Kubernetes API [started]
       › Current Kubernetes context: 'default/api-ocp410-aws-crw-qe-com:6443/admin'
       [08:32:41] Verify Kubernetes API...[1.23] [title changed]
       [08:32:41] Verify Kubernetes API...[1.23] [completed]
       [08:32:41] OpenShift version [started]
       [08:32:41] OpenShift version...[4.10] [title changed]
       [08:32:41] OpenShift version...[4.10] [completed]
       [08:32:41] Openshift preflight checklist [started]
       [08:32:41] Verify if oc is installed [started]
       [08:32:41] Verify if oc is installed...[OK] [title changed]
       [08:32:41] Verify if oc is installed...[OK] [completed]
       [08:32:41] Verify if openshift is running [started]
       [08:32:42] Verify if openshift is running...[OK] [title changed]
       [08:32:42] Verify if openshift is running...[OK] [completed]
       [08:32:42] Openshift preflight checklist [completed]
       [08:32:42] Create Namespace openshift-devspaces [started]
       [08:32:43] Create Namespace openshift-devspaces...[Exists] [title changed]
       [08:32:43] Create Namespace openshift-devspaces...[Exists] [completed]
       [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs [started]
       [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs...[OK] [title changed]
       [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs...[OK] [completed]
       [08:32:43] Deploy Red Hat OpenShift Dev Spaces [started]
       [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable [started]
       [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable...[Exists] [title changed]
       [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable...[Exists] [completed]
       [08:32:43] Install Dev Workspace operator [started]
       [08:32:43] Create Subscription devworkspace-operator [started]
       [08:32:45] Create Subscription devworkspace-operator...[Exists] [title changed]
       [08:32:45] Create Subscription devworkspace-operator...[Exists] [completed]
       [08:32:45] Wait for Dev Workspace operator ready [started]
       [08:34:19] Wait for Dev Workspace operator ready [failed]
       [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
       [08:34:19] Install Dev Workspace operator [failed]
       [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
       [08:34:19] Deploy Red Hat OpenShift Dev Spaces [failed]
       [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
           Error: Command server:deploy failed with the error: ERR_TIMEOUT: Timeout 
           set to pod ready timeout 60000 See details: 
           /home/hudson/.cache/dsc/error.log. Red Hat OpenShift Dev Spaces logs: 
           /tmp/dsc-logs/1688387559944.
      

      Root cause: DevWorkspace Operator installation failure:

      Failed to pull image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec": rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found
      

      The problem has been reproduced locally:

      $ podman pull brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec
      Trying to pull brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec...
      Error: determining manifest MIME type for docker://brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found
      

      Solution: don't mirror registry.redhat.io/openshift4/ose-kube-rbac-proxy when deploy test devspace instances using CRW Jenkins test pipeline:

      oc apply -f - <<EOF
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
        name: image-registry
      spec:
        repositoryDigestMirrors:
          - mirrors:
              - brew.registry.redhat.io/devspaces
            source: registry.redhat.io/devspaces
          - mirrors:
              - brew.registry.redhat.io/devworkspace
            source: registry.redhat.io/devworkspace
          - mirrors:
              - brew.registry.redhat.io/devspaces
            source: registry.stage.redhat.io/devspaces
          - mirrors:
              - brew.registry.redhat.io/devworkspace
            source: registry.stage.redhat.io/devworkspace
          - mirrors:
              - brew.registry.redhat.io
            source: registry-proxy.engineering.redhat.com
      EOF
      

            dnochevn Dmytro Nochevnov
            dnochevn Dmytro Nochevnov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: