Uploaded image for project: 'Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces) '
  1. Red Hat OpenShift Dev Spaces (formerly CodeReady Workspaces)
  2. CRW-1930

CRW 2.9.0.RC-06-16 installation failed on OCP 4.8.0-RC0 on postgres deployment step

XMLWordPrintable

    • False
    • False
    • Undefined

      crwctl crwctl/2.9.0-CI-redhat.0e1e643 linux-x64 node-v12.21.0 failed to install CRW 2.9.0.RC-06-16 on QE OCP 4.8.0-RC0 on AWS to project https://console-openshift-console.apps.ocp48-aws.crw-qe.com/k8s/cluster/projects/install-crw-using-crwctl:

      https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/CodereadyWorkspaces/job/e2e/job/basic/job/install-crw-using-crwctl/635/consoleFull

      crwctl logs
      [Pipeline] sh
       + /mnt/hudson_workspace/workspace/CodereadyWorkspaces/e2e/basic/install-crw-using-crwctl/crwctl/bin/crwctl server:deploy --no-olm-suggested-namespace --catalog-source-name=osbs-catalog-source --catalog-source-namespace=openshift-marketplace --package-manifest-name=codeready-workspaces --olm-channel=latest --listr-renderer=verbose -n install-crw-using-crwctl
       (node:11003) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
       › Installer type is set to: 'olm'
       [11:14:21] Verify Kubernetes API [started]
       › Current Kubernetes context: 'default/api-ocp48-aws-crw-qe-com:6443/admin'
       [11:14:21] Verify Kubernetes API...OK [title changed]
       [11:14:22] Verify Kubernetes API...OK (it's OpenShift) [title changed]
       [11:14:22] Verify Kubernetes API...OK (it's OpenShift) [completed]
       [11:14:22]   Looking for an already existing CodeReady Workspaces instance [started]
       [11:14:22] Verify if CodeReady Workspaces is deployed into namespace "install-crw-using-crwctl" [started]
       [11:14:22] Verify if CodeReady Workspaces is deployed into namespace "install-crw-using-crwctl"...it is not [title changed]
       [11:14:22] Verify if CodeReady Workspaces is deployed into namespace "install-crw-using-crwctl"...it is not [completed]
       [11:14:22]   Looking for an already existing CodeReady Workspaces instance [completed]
       [11:14:22]   Openshift preflight checklist [started]
       [11:14:22] Verify if oc is installed [started]
       [11:14:22] Verify if oc is installed...done. [title changed]
       [11:14:22] Verify if oc is installed...done. [completed]
       [11:14:22] Verify if openshift is running [started]
       [11:14:23] Verify if openshift is running...done. [title changed]
       [11:14:23] Verify if openshift is running...done. [completed]
       [11:14:23] Check OpenShift version [started]
       [11:14:24] Check OpenShift version: 4.x [title changed]
       [11:14:24] Check OpenShift version: 4.x [completed]
       [11:14:24] Check Kubernetes version [started]
       [11:14:24] Check Kubernetes version: Unknown. [title changed]
       [11:14:24] Check Kubernetes version: Unknown. [completed]
       [11:14:24]   Openshift preflight checklist [completed]
       [11:14:24] Following CodeReady Workspaces logs [started]
       [11:14:24] Start following Operator logs [started]
       [11:14:24] Start following Operator logs...done [title changed]
       [11:14:24] Start following Operator logs...done [completed]
       [11:14:24] Start following CodeReady Workspaces Server logs [started]
       [11:14:24] Start following CodeReady Workspaces Server logs...done [title changed]
       [11:14:24] Start following CodeReady Workspaces Server logs...done [completed]
       [11:14:24] Start following PostgreSQL logs [started]
       [11:14:24] Start following PostgreSQL logs...done [title changed]
       [11:14:24] Start following PostgreSQL logs...done [completed]
       [11:14:24] Start following Keycloak logs [started]
       [11:14:24] Start following Keycloak logs...done [title changed]
       [11:14:24] Start following Keycloak logs...done [completed]
       [11:14:24] Start following Plug-in Registry logs [started]
       [11:14:24] Start following Plug-in Registry logs...done [title changed]
       [11:14:24] Start following Plug-in Registry logs...done [completed]
       [11:14:24] Start following Devfile Registry logs [started]
       [11:14:24] Start following Devfile Registry logs...done [title changed]
       [11:14:24] Start following Devfile Registry logs...done [completed]
       [11:14:24] Start following CodeReady Workspaces Dashboard logs [started]
       [11:14:24] Start following CodeReady Workspaces Dashboard logs...done [title changed]
       [11:14:24] Start following CodeReady Workspaces Dashboard logs...done [completed]
       [11:14:24] Start following namespace events [started]
       [11:14:24] Start following namespace events...done [title changed]
       [11:14:24] Start following namespace events...done [completed]
       [11:14:24] Following CodeReady Workspaces logs [completed]
       [11:14:24]   Running Olm installaion CodeReady Workspaces [started]
       [11:14:24] Check if OLM is pre-installed on the platform [started]
       [11:14:24] Check if OLM is pre-installed on the platform...done. [title changed]
       [11:14:24] Check if OLM is pre-installed on the platform...done. [completed]
       [11:14:24] Create Namespace (install-crw-using-crwctl) [started]
       [11:14:25] Create Namespace (install-crw-using-crwctl)...Done. [title changed]
       [11:14:25] Create Namespace (install-crw-using-crwctl)...Done. [completed]
       [11:14:25] Create operator group [started]
       [11:14:25] Create operator group...created new one. [title changed]
       [11:14:25] Create operator group...created new one. [completed]
       [11:14:25] Configure context information [started]
       [11:14:25] Configure context information...done. [title changed]
       [11:14:25] Configure context information...done. [completed]
       [11:14:25] Create operator subscription [started]
       [11:14:26] Create operator subscription...created new one. [title changed]
       [11:14:26] Create operator subscription...created new one. [completed]
       [11:14:26] Wait while subscription is ready [started]
       [11:14:34] Wait while subscription is ready...done. [title changed]
       [11:14:34] Wait while subscription is ready...done. [completed]
       [11:14:34] Wait while operator installed [started]
       [11:14:40] Wait while operator installed...done. [title changed]
       [11:14:40] Wait while operator installed...done. [completed]
       [11:14:40] Prepare CodeReady Workspaces cluster CR [started]
       [11:14:41] Prepare CodeReady Workspaces cluster CR...Done. [title changed]
       [11:14:41] Prepare CodeReady Workspaces cluster CR...Done. [completed]
       [11:14:41] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace install-crw-using-crwctl [started]
       [11:14:42] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace install-crw-using-crwctl...done. [title changed]
       [11:14:42] Create the Custom Resource of type checlusters.org.eclipse.che in the namespace install-crw-using-crwctl...done. [completed]
       [11:14:42]   Running Olm installaion CodeReady Workspaces [completed]
       [11:14:42]   Post installation checklist [started]
       [11:14:42] PostgreSQL pod bootstrap [started]
       [11:14:42] Scheduling [started]
       [11:15:09] Scheduling...done [title changed]
       [11:15:09] Scheduling...done [completed]
       [11:15:09] Downloading images [started]
       [11:15:17] Downloading images...done [title changed]
       [11:15:17] Downloading images...done [completed]
       [11:15:17] Starting [started]
       [11:15:39] Starting...done [title changed]
       [11:15:39] Starting...done [completed]
       [11:15:39] PostgreSQL pod bootstrap [completed]
       [11:15:39] Keycloak pod bootstrap [started]
       [11:15:39] Scheduling [started]
       [11:34:30] Scheduling [failed]
       [11:34:30]  Timeout: there are no pods in the namespace: install-crw-using-crwctl, selector: app=codeready,component=keycloak. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
       [11:34:30] Keycloak pod bootstrap [failed]
       [11:34:30]  Timeout: there are no pods in the namespace: install-crw-using-crwctl, selector: app=codeready,component=keycloak. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
       [11:34:30]   Post installation checklist [failed]
       [11:34:30]  Timeout: there are no pods in the namespace: install-crw-using-crwctl, selector: app=codeready,component=keycloak. Check CodeReady Workspaces logs for details. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
        ›   Error: Error: Timeout: there are no pods in the namespace: 
        ›   install-crw-using-crwctl, selector: app=codeready,component=keycloak. 
        ›   Check CodeReady Workspaces logs for details. Consider increasing error 
        ›   recheck timeout with --k8spoderrorrechecktimeout flag.
        ›   Command server:deploy failed. Error log: 
        ›   /home/hudson/.cache/crwctl/error.log
      

       
      https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/CodereadyWorkspaces/job/e2e/job/basic/job/install-crw-using-crwctl/635/artifact/logs-and-configs/logs/codeready-operator-d94b6bc4b-nh7bq.pod.log

      operator logs
      time="2021-06-17T15:15:47Z" level=info msg="Running exec for 'create Keycloak DB, user, privileges' in the pod 'postgres-7fb5f47796-csck5'"
      time="2021-06-17T15:16:00Z" level=error msg="Error running exec: Internal error occurred: admission plugin \"ValidatingAdmissionWebhook\" failed to complete validation in 13s, command: [/bin/bash -c OUT=$(psql postgres -tAc \"SELECT 1 FROM pg_roles WHERE rolname='keycloak'\"); if [ $OUT -eq 1 ]; then echo \"DB exists\"; exit 0; fi && psql -c \"CREATE USER keycloak WITH PASSWORD 'SEhfneqMbw2L'\" && psql -c \"CREATE DATABASE keycloak\" && psql -c \"GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak\" && psql -c \"ALTER USER ${POSTGRESQL_USER} WITH SUPERUSER\"]"
      time="2021-06-17T15:16:00Z" level=error msg="Stderr: "
      

      https://main-jenkins-csb-crwqe.apps.ocp4.prod.psi.redhat.com/job/CodereadyWorkspaces/job/e2e/job/basic/job/install-crw-using-crwctl/635/artifact/logs-and-configs/logs/postgres-7fb5f47796-csck5.pod.log

      Postgres pod logs
      The files belonging to this database system will be owned by user "postgres".
      This user must also own the server process.
      
      The database cluster will be initialized with locale "en_US.utf8".
      The default database encoding has accordingly been set to "UTF8".
      The default text search configuration will be set to "english".
      
      Data page checksums are disabled.
      
      fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
      creating subdirectories ... ok
      selecting default max_connections ... 100
      selecting default shared_buffers ... 128MB
      selecting default timezone ... Etc/UTC
      selecting dynamic shared memory implementation ... posix
      creating configuration files ... ok
      running bootstrap script ... ok
      performing post-bootstrap initialization ... ok
      syncing data to disk ... ok
      
      Success. You can now start the database server using:
      
          pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start
      
      
      WARNING: enabling "trust" authentication for local connections
      You can change this by editing pg_hba.conf or using the option -A, or
      --auth-local and --auth-host, the next time you run initdb.
      waiting for server to start....LOG:  redirecting log output to logging collector process
      HINT:  Future log output will appear in directory "pg_log".
       done
      server started
      /var/run/postgresql:5432 - accepting connections
      => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
      ALTER ROLE
      ALTER ROLE
      waiting for server to shut down.... done
      server stopped
      Starting server...
      LOG:  redirecting log output to logging collector process
      HINT:  Future log output will appear in directory "pg_log".
      

      Postgres image: registry.redhat.io/rhel8/postgresql-96@sha256:1088070ff2f1404a312485de2ac3aaa8360539d0a0e59429520bde6b2facd5d6

      Update: there was `crwctl server:delete` command executed before installation.
       

            dnochevn Dmytro Nochevnov
            dnochevn Dmytro Nochevnov
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: