Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-1124

quay.io/crw/server-rhel8:2.3-37 deployment failed because of "chmod: changing permissions of '/home/user/cacerts': Operation not permitted" error

    XMLWordPrintable

Details

    Description

      CRW 2.3.0.ER-07-30 codeready deployment failed during installation by crwctl 2.3.0-CI-6129bc8 on QE OCP 3.11/4.4 because of error:

      https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/install-crw-using-crwctl/1420/

      codeready deployment log
      Using custom assembly from /home/user/codeready
Found a custom cert. Adding it to java trust store /home/user/cacerts
chmod: changing permissions of '/home/user/cacerts': Operation not permitted
      command to install CRW on OCP 3.11
      crwctl/bin/crwctl server:start --che-operator-cr-patch-yaml=custom-resource-patch.yaml --che-operator-image=quay.io/crw/crw-2-rhel8-operator:latest --listr-renderer=verbose --k8spodreadytimeout=300000 -n crw-crwctl-tls-oauth
      command to install CRW on OCP 4.4
      crwctl/bin/crwctl server:start --che-operator-cr-patch-yaml=custom-resource-patch.yaml --catalog-source-name=codeready-workspaces-latest --catalog-source-namespace=openshift-marketplace --package-manifest-name=codeready-workspaces --olm-channel=latest --listr-renderer=verbose --k8spodreadytimeout=300000 -n crw-crwctl-with-tls-no-oauth

      where

      custom-resource-patch.yaml
      spec:
  server:
    cheImage: 'quay.io/crw/server-rhel8'
    cheImageTag: 'latest'
    devfileRegistryImage: 'quay.io/crw/devfileregistry-rhel8:latest'
    pluginRegistryImage: 'quay.io/crw/pluginregistry-rhel8:latest'
    customCheProperties:
      CHE_WORKSPACE_PLUGIN__BROKER_METADATA_IMAGE: 'quay.io/crw/pluginbroker-metadata-rhel8:latest'
      CHE_WORKSPACE_PLUGIN__BROKER_ARTIFACTS_IMAGE: 'quay.io/crw/pluginbroker-artifacts-rhel8:latest'
      CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: 'quay.io/crw/jwtproxy-rhel8:latest'
      CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: '-1'
      CHE_LIMITS_WORKSPACE_IDLE_TIMEOUT: '900000'
      CHE_DOCKER_ALWAYS__PULL__IMAGE: 'true'
  auth:
    openShiftoAuth: true
    updateAdminPassword: false

      quay.io/crw/server-rhel8:latest == quay.io/crw/server-rhel8:2.3-37

      crwctl logs
      10:05:43 › Current Kubernetes context: 'default/console-ocp311-crw:8443/admin'
10:05:43 › Installer type is set to: 'operator'
10:05:43 [03:05:43] Verify Kubernetes API [started]
10:05:43 [03:05:43] Verify Kubernetes API...OK [title changed]
10:05:43 [03:05:43] Verify Kubernetes API...OK (it's OpenShift) [title changed]
10:05:43 [03:05:43] Verify Kubernetes API...OK (it's OpenShift) [completed]
10:05:43 [03:05:43]   Looking for an already existing CodeReady Workspaces instance [started]
10:05:43 [03:05:43] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl-tls-oauth" [started]
10:05:43 [03:05:43] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl-tls-oauth"...it is not [title changed]
10:05:43 [03:05:43] Verify if CodeReady Workspaces is deployed into namespace "crw-crwctl-tls-oauth"...it is not [completed]
10:05:43 [03:05:43]   Looking for an already existing CodeReady Workspaces instance [completed]
10:05:43 [03:05:43]   Openshift preflight checklist [started]
10:05:43 [03:05:43] Verify if oc is installed [started]
10:05:43 [03:05:43] Verify if oc is installed...done. [title changed]
10:05:43 [03:05:43] Verify if oc is installed...done. [completed]
10:05:43 [03:05:43] Verify if openshift is running [started]
10:05:43 [03:05:43] Verify if openshift is running...done. [title changed]
10:05:43 [03:05:43] Verify if openshift is running...done. [completed]
10:05:43 [03:05:43] Check OpenShift version [started]
10:05:44 [03:05:43] Check OpenShift version: v3.11.248. [title changed]
10:05:44 [03:05:43] Check OpenShift version: v3.11.248. [completed]
10:05:44 [03:05:43] Check Kubernetes version [started]
10:05:44 [03:05:44] Check Kubernetes version: Found v1.11.0+d4cacc0. [title changed]
10:05:44 [03:05:44] Check Kubernetes version: Found v1.11.0+d4cacc0. [completed]
10:05:44 [03:05:44]   Openshift preflight checklist [completed]
10:05:44 [03:05:44] Verify Openshift oauth. [started]
10:05:44 [03:05:44] Verify Openshift oauth....done. [title changed]
10:05:44 [03:05:44] Verify Openshift oauth....done. [completed]
10:05:44 CodeReady Workspaces logs will be available in '/tmp/crwctl-logs/1596092743411'
10:05:44 [03:05:44] Start following logs [started]
10:05:44 [03:05:44] Start following Operator logs [started]
10:05:44 [03:05:44] Start following Operator logs...done [title changed]
10:05:44 [03:05:44] Start following Operator logs...done [completed]
10:05:44 [03:05:44] Start following CodeReady Workspaces logs [started]
10:05:44 [03:05:44] Start following CodeReady Workspaces logs...done [title changed]
10:05:44 [03:05:44] Start following CodeReady Workspaces logs...done [completed]
10:05:44 [03:05:44] Start following Postgres logs [started]
10:05:44 [03:05:44] Start following Postgres logs...done [title changed]
10:05:44 [03:05:44] Start following Postgres logs...done [completed]
10:05:44 [03:05:44] Start following Keycloak logs [started]
10:05:44 [03:05:44] Start following Keycloak logs...done [title changed]
10:05:44 [03:05:44] Start following Keycloak logs...done [completed]
10:05:44 [03:05:44] Start following Plugin registry logs [started]
10:05:44 [03:05:44] Start following Plugin registry logs...done [title changed]
10:05:44 [03:05:44] Start following Plugin registry logs...done [completed]
10:05:44 [03:05:44] Start following Devfile registry logs [started]
10:05:44 [03:05:44] Start following Devfile registry logs...done [title changed]
10:05:44 [03:05:44] Start following Devfile registry logs...done [completed]
10:05:44 [03:05:44] Start following logs [completed]
10:05:44 [03:05:44] Start following events [started]
10:05:44 [03:05:44] Start following namespace events [started]
10:05:44 [03:05:44] Start following namespace events...done [title changed]
10:05:44 [03:05:44] Start following namespace events...done [completed]
10:05:44 [03:05:44] Start following events [completed]
10:05:44 [03:05:44]   Running the CodeReady Workspaces operator [started]
10:05:44 [03:05:44] Copying operator resources [started]
10:05:44 [03:05:44] Copying operator resources...done. [title changed]
10:05:44 [03:05:44] Copying operator resources...done. [completed]
10:05:44 [03:05:44] Create Namespace (crw-crwctl-tls-oauth) [started]
10:05:44 [03:05:44] Create Namespace (crw-crwctl-tls-oauth)...done. [title changed]
10:05:44 [03:05:44] Create Namespace (crw-crwctl-tls-oauth)...done. [completed]
10:05:44 [03:05:44] Create ServiceAccount codeready-operator in namespace crw-crwctl-tls-oauth [started]
10:05:44 [03:05:44] Create ServiceAccount codeready-operator in namespace crw-crwctl-tls-oauth...done. [title changed]
10:05:44 [03:05:44] Create ServiceAccount codeready-operator in namespace crw-crwctl-tls-oauth...done. [completed]
10:05:44 [03:05:44] Create Role codeready-operator in namespace crw-crwctl-tls-oauth [started]
10:05:44 [03:05:44] Create Role codeready-operator in namespace crw-crwctl-tls-oauth...done. [title changed]
10:05:44 [03:05:44] Create Role codeready-operator in namespace crw-crwctl-tls-oauth...done. [completed]
10:05:44 [03:05:44] Create ClusterRole codeready-operator [started]
10:05:44 [03:05:44] Create ClusterRole codeready-operator...done. [title changed]
10:05:44 [03:05:44] Create ClusterRole codeready-operator...done. [completed]
10:05:44 [03:05:44] Create RoleBinding codeready-operator in namespace crw-crwctl-tls-oauth [started]
10:05:45 [03:05:44] Create RoleBinding codeready-operator in namespace crw-crwctl-tls-oauth...done. [title changed]
10:05:45 [03:05:44] Create RoleBinding codeready-operator in namespace crw-crwctl-tls-oauth...done. [completed]
10:05:45 [03:05:44] Create ClusterRoleBinding codeready-operator [started]
10:05:45 [03:05:44] Create ClusterRoleBinding codeready-operator...done. [title changed]
10:05:45 [03:05:44] Create ClusterRoleBinding codeready-operator...done. [completed]
10:05:45 [03:05:44] Create CRD checlusters.org.eclipse.che [started]
10:05:45 [03:05:44] Create CRD checlusters.org.eclipse.che...done. [title changed]
10:05:45 [03:05:44] Create CRD checlusters.org.eclipse.che...done. [completed]
10:05:45 [03:05:44] Waiting 5 seconds for the new Kubernetes resources to get flushed [started]
10:05:50 [03:05:49] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [title changed]
10:05:50 [03:05:49] Waiting 5 seconds for the new Kubernetes resources to get flushed...done. [completed]
10:05:50 [03:05:49] Create deployment codeready-operator in namespace crw-crwctl-tls-oauth [started]
10:05:50 [03:05:49] Create deployment codeready-operator in namespace crw-crwctl-tls-oauth...done. [title changed]
10:05:50 [03:05:49] Create deployment codeready-operator in namespace crw-crwctl-tls-oauth...done. [completed]
10:05:50 [03:05:49] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl-tls-oauth [started]
10:05:50 [03:05:49] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl-tls-oauth...done. [title changed]
10:05:50 [03:05:49] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace crw-crwctl-tls-oauth...done. [completed]
10:05:50 [03:05:49]   Running the CodeReady Workspaces operator [completed]
10:05:50 [03:05:49]   Post installation checklist [started]
10:05:50 [03:05:49] PostgreSQL pod bootstrap [started]
10:05:50 [03:05:49] scheduling [started]
10:06:02 [03:06:00] scheduling...done. [title changed]
10:06:02 [03:06:00] scheduling...done. [completed]
10:06:02 [03:06:00] downloading images [started]
10:06:20 [03:06:18] downloading images...done. [title changed]
10:06:20 [03:06:18] downloading images...done. [completed]
10:06:20 [03:06:18] starting [started]
10:06:47 [03:06:43] starting...done. [title changed]
10:06:47 [03:06:43] starting...done. [completed]
10:06:47 [03:06:43] PostgreSQL pod bootstrap [completed]
10:06:47 [03:06:43] Keycloak pod bootstrap [started]
10:06:47 [03:06:43] scheduling [started]
10:06:47 [03:06:45] scheduling...done. [title changed]
10:06:47 [03:06:45] scheduling...done. [completed]
10:06:47 [03:06:45] downloading images [started]
10:07:13 [03:07:12] downloading images...done. [title changed]
10:07:13 [03:07:12] downloading images...done. [completed]
10:07:13 [03:07:12] starting [started]
10:07:52 [03:07:51] starting...done. [title changed]
10:07:52 [03:07:51] starting...done. [completed]
10:07:52 [03:07:51] Keycloak pod bootstrap [completed]
10:07:52 [03:07:51] Devfile registry pod bootstrap [started]
10:07:52 [03:07:51] scheduling [started]
10:09:43 [03:09:32] scheduling...done. [title changed]
10:09:43 [03:09:32] scheduling...done. [completed]
10:09:43 [03:09:32] downloading images [started]
10:09:51 [03:09:50] downloading images...done. [title changed]
10:09:51 [03:09:50] downloading images...done. [completed]
10:09:51 [03:09:50] starting [started]
10:09:53 [03:09:53] starting...done. [title changed]
10:09:53 [03:09:53] starting...done. [completed]
10:09:53 [03:09:53] Devfile registry pod bootstrap [completed]
10:09:53 [03:09:53] Plugin registry pod bootstrap [started]
10:09:53 [03:09:53] scheduling [started]
10:09:53 [03:09:53] scheduling...done. [title changed]
10:09:53 [03:09:53] scheduling...done. [completed]
10:09:53 [03:09:53] downloading images [started]
10:10:49 [03:10:46] downloading images...done. [title changed]
10:10:49 [03:10:46] downloading images...done. [completed]
10:10:49 [03:10:46] starting [started]
10:10:56 [03:10:56] starting...done. [title changed]
10:10:56 [03:10:56] starting...done. [completed]
10:10:56 [03:10:56] Plugin registry pod bootstrap [completed]
10:10:56 [03:10:56] CodeReady Workspaces pod bootstrap [started]
10:10:56 [03:10:56] scheduling [started]
10:10:56 [03:10:56] scheduling...done. [title changed]
10:10:56 [03:10:56] scheduling...done. [completed]
10:10:56 [03:10:56] downloading images [started]
10:11:14 [03:11:11] downloading images...done. [title changed]
10:11:14 [03:11:11] downloading images...done. [completed]
10:11:14 [03:11:11] starting [started]
10:16:21 [03:16:18] starting [failed]
10:16:21 [03:16:18]  ERR_TIMEOUT: Timeout set to pod ready timeout 300000
10:16:21 [03:16:18] CodeReady Workspaces pod bootstrap [failed]
10:16:21 [03:16:18]  ERR_TIMEOUT: Timeout set to pod ready timeout 300000
10:16:21 [03:16:18]   Post installation checklist [failed]
10:16:21 [03:16:18]  ERR_TIMEOUT: Timeout set to pod ready timeout 300000
10:16:21  ›   Error: Error: ERR_TIMEOUT: Timeout set to pod ready timeout 300000
10:16:21  ›   Installation failed, check logs in '/tmp/crwctl-logs/1596092743411'

       

      Attachments

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. CRW-1093 Adding certificates to the server truststore ConfigMap doesn't work

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. CRW-1189 "chmod: changing permissions of '/home/user/cacerts': Operation not permitted" error of quay.io/crw/server-rhel8:2.4-4 server container start

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. CRW-1098 "/bin/sh: /opt/eap/che.crt: No such file or directory" keycloak deployment error when install CRW 2.3.0 on OCP 3.11

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          relates to

          Task - Represents a small unit of work that is not end-user facing. CRW-1381 Synchronize Che-server docker image upstream-downstream work

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Activity

            People

              nickboldt Nick Boldt
              dnochevn Dmytro Nochevnov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: