Loading...

Type: Task
Resolution: Done
Priority: Blocker
Fix Version/s: 3.7.0.GA
Affects Version/s: 3.7.0.GA
Component/s: Team A: Dashboard, devfile and plugin registries, open-vsx, operator + chectl/dsc, devfile-converter, configbump + traefik, image-puller, server, gateway, authentication, try-in-web-ide action, telemetry
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False
Intelligence Requested:
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

There was Dev Spaces 3.7.0 deployment error:
https://main-jenkins-csb-crwqe.apps.ocp-c1.prod.psi.redhat.com/job/Testing/job/e2e/job/basic/job/install-ds/2543/console

dsc logs

 + /mnt/hudson_workspace/workspace/Testing/e2e/basic/install-ds/dsc/bin/dsc server:deploy --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/Testing/e2e/basic/install-ds/custom-resource-patch.yaml --package-manifest-name=devspaces --catalog-source-name=devspaces-operator-3-7-osbs-stable --olm-channel=stable --catalog-source-namespace=openshift-marketplace --listr-renderer=verbose
 [08:32:41] Verify Kubernetes API [started]
 › Current Kubernetes context: 'default/api-ocp410-aws-crw-qe-com:6443/admin'
 [08:32:41] Verify Kubernetes API...[1.23] [title changed]
 [08:32:41] Verify Kubernetes API...[1.23] [completed]
 [08:32:41] OpenShift version [started]
 [08:32:41] OpenShift version...[4.10] [title changed]
 [08:32:41] OpenShift version...[4.10] [completed]
 [08:32:41] Openshift preflight checklist [started]
 [08:32:41] Verify if oc is installed [started]
 [08:32:41] Verify if oc is installed...[OK] [title changed]
 [08:32:41] Verify if oc is installed...[OK] [completed]
 [08:32:41] Verify if openshift is running [started]
 [08:32:42] Verify if openshift is running...[OK] [title changed]
 [08:32:42] Verify if openshift is running...[OK] [completed]
 [08:32:42] Openshift preflight checklist [completed]
 [08:32:42] Create Namespace openshift-devspaces [started]
 [08:32:43] Create Namespace openshift-devspaces...[Exists] [title changed]
 [08:32:43] Create Namespace openshift-devspaces...[Exists] [completed]
 [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs [started]
 [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs...[OK] [title changed]
 [08:32:43] Start following Red Hat OpenShift Dev Spaces installation logs...[OK] [completed]
 [08:32:43] Deploy Red Hat OpenShift Dev Spaces [started]
 [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable [started]
 [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable...[Exists] [title changed]
 [08:32:43] Create CatalogSource devspaces-operator-3-7-osbs-stable...[Exists] [completed]
 [08:32:43] Install Dev Workspace operator [started]
 [08:32:43] Create Subscription devworkspace-operator [started]
 [08:32:45] Create Subscription devworkspace-operator...[Exists] [title changed]
 [08:32:45] Create Subscription devworkspace-operator...[Exists] [completed]
 [08:32:45] Wait for Dev Workspace operator ready [started]
 [08:34:19] Wait for Dev Workspace operator ready [failed]
 [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
 [08:34:19] Install Dev Workspace operator [failed]
 [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
 [08:34:19] Deploy Red Hat OpenShift Dev Spaces [failed]
 [08:34:19] → ERR_TIMEOUT: Timeout set to pod ready timeout 60000
     Error: Command server:deploy failed with the error: ERR_TIMEOUT: Timeout 
     set to pod ready timeout 60000 See details: 
     /home/hudson/.cache/dsc/error.log. Red Hat OpenShift Dev Spaces logs: 
     /tmp/dsc-logs/1688387559944.

Root cause: DevWorkspace Operator installation failure:

Failed to pull image "registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec": rpc error: code = Unknown desc = fetching target platform image selected from manifest list: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found

The problem has been reproduced locally:

$ podman pull brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec
Trying to pull brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec...
Error: determining manifest MIME type for docker://brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy@sha256:b1dcc377a56d26dd0fc4fc360b45b515f5cf8b368f5f81540f5f047447d17fec: reading manifest sha256:1941f88bcd73332cba3f794ee2dd3815a2255c9adfbbab4d4aa71f1710e2f614 in brew.registry.redhat.io/openshift4/ose-kube-rbac-proxy: name unknown: Digest not found

Solution: don't mirror registry.redhat.io/openshift4/ose-kube-rbac-proxy when deploy test devspace instances using CRW Jenkins test pipeline:

oc apply -f - <<EOF
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: image-registry
spec:
  repositoryDigestMirrors:
    - mirrors:
        - brew.registry.redhat.io/devspaces
      source: registry.redhat.io/devspaces
    - mirrors:
        - brew.registry.redhat.io/devworkspace
      source: registry.redhat.io/devworkspace
    - mirrors:
        - brew.registry.redhat.io/devspaces
      source: registry.stage.redhat.io/devspaces
    - mirrors:
        - brew.registry.redhat.io/devworkspace
      source: registry.stage.redhat.io/devworkspace
    - mirrors:
        - brew.registry.redhat.io
      source: registry-proxy.engineering.redhat.com
EOF