Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-7001

OMR 1.3.11 upgrade from 1.3.10 was failed at the step of "Wait for Quay"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Blocker Blocker
    • omr-v1.3.11
    • omr-v1.3.11
    • OMR
    • False
    • None
    • False

      Description:

      This is an issue found in OMR 1.3.11 upgrade testing, after triggered OMR 1.3.11 upgrade from OMR 1.3.10, the OMR upgrade was failed at step "TASK [mirror_appliance : Wait for Quay]", checked the logs, get error message with "Error: running container create option: no secret with name or id "pgdb_pass": no such secret", "Error: running container create option: no secret with name or id "redis_pass": no such secret", pls review attached logs omr.logs 

      OMR:  openshift-mirror-registry-rhel8:v1.3.11-5

      OMR Upgrade Logs:

      ./mirror-registry --version   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containersmirror-registry version v1.3.11

./mirror-registry upgrade -H quayomr.qe.gcp.devcluster.openshift.com   __   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your ContainersINFO[2024-04-11 07:22:18] Upgrade has begun                            
INFO[2024-04-11 07:22:18] Found execution environment at /home/lizhang/execution-environment.tar 
INFO[2024-04-11 07:22:18] Loading execution environment from execution-environment.tar 
INFO[2024-04-11 07:22:22] Found SSH key at /home/lizhang/.ssh/quay_installer 
INFO[2024-04-11 07:22:22] Attempting to set SELinux rules on /home/lizhang/.ssh/quay_installer 
INFO[2024-04-11 07:22:22] Found image archive at /home/lizhang/image-archive.tar 
INFO[2024-04-11 07:22:22] Attempting to set SELinux rules on image archive 
INFO[2024-04-11 07:22:22] Running upgrade playbook. This may take some time. To see playbook output run the installer with -v (verbose) flag. PLAY [Upgrade Mirror Appliance] *******************************************************************************************************************************TASK [Gathering Facts] ****************************************************************************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand variables] ********************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/expand-vars.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Expand pg_storage] *******************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand quay_root] ********************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand quay_storage] *****************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Set expanded variables] **************************************************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Install Dependencies] ****************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/install-deps.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Create user service directory] *******************************************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Set SELinux Rules] *******************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/set-selinux-rules.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Set container_manage_cgroup flag on and keep it persistent across reboots] ***********************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Autodetect root or rootless from previous install] ***********************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/migrate-pre-1_3_0.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Check for old quay-storage] **********************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Stop Quay service] *******************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Create temporary data-only container for quay-storage] *******************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy data to quay-storage volume] ****************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete temporary container] **********************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete /etc/quay-install/quay-storage] ***********************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check for old pg-storage] ************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Stop Postgres service] ***************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Create temporary data-only container for pg-storage] *********************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy data to pg-storage volume] ******************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Fix permissions] *********************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete temporary container] **********************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete /etc/quay-install/pg-data] ****************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check for old Quay root /etc/quay-install] *******************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Migrate Quay root] *******************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Autodetect Image Archive] ************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/autodetect-image-archive.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Checking for Image Archive] **********************************************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com -> localhost]TASK [mirror_appliance : Create install directory for image-archive.tar dest] *********************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy Images if /runner/image-archive.tar exists] *************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Unpack Images if /runner/image-archive.tar exists] ***********************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Redis if redis.tar exists] ***************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Quay if quay.tar exists] *****************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Postgres if postgres.tar exists] *********************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Quay Pod Service] ************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/upgrade-pod-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Quay Pod systemd service file] **************************************************************************************************
ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if pod pause image is loaded] **************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Infra image] ********************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Quay Pod service] **************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Postgres Service] ************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/upgrade-postgres-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Postgres systemd service file] **************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Postgres image is loaded] ***************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Postgres image] *****************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Postgres service] **************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Redis Service] ***************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/upgrade-redis-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Redis systemd service file] *****************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Redis image is loaded] ******************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Redis image] ********************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Redis service] *****************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Quay Service] ****************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/upgrade-quay-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Quay systemd service file] ******************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Quay image is loaded] *******************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Quay image] *********************************************************************************************************************
skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Quay service] ******************************************************************************************************************
changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Wait for Quay] ***********************************************************************************************************************
included: /runner/project/roles/mirror_appliance/tasks/wait-for-quay.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance] ************
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (10 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (9 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (8 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (7 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (6 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (5 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (4 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (3 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (2 retries left).
FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (1 retries left).
fatal: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 104] Connection reset by peer>", "redirected": false, "status": -1, "url": "https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance"}PLAY RECAP ****************************************************************************************************************************************************
lizhang@quayomr.qe.gcp.devcluster.openshift.com : ok=35   changed=19   unreachable=0    failed=1    skipped=20   rescued=0    ignored=0   ERRO[2024-04-11 07:31:10] An error occurred: exit status 2   

      OMR upgrade errors:

      Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Service RestartSec=30s expired, scheduling restart.
Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Scheduled restart job, restart counter is at 29.
Apr 11 07:40:06 quay311omr systemd[1595]: Stopped PostgreSQL Podman Container for Quay.
Apr 11 07:40:06 quay311omr systemd[1595]: Starting PostgreSQL Podman Container for Quay...
Apr 11 07:40:06 quay311omr systemd[1595]: Started PostgreSQL Podman Container for Quay.
Apr 11 07:40:06 quay311omr podman[34198]: Error: running container create option: no secret with name or id "pgdb_pass": no such secret
Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Main process exited, code=exited, status=125/n/a
Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Failed with result 'exit-code'.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Service RestartSec=30s expired, scheduling restart.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Scheduled restart job, restart counter is at 29.
Apr 11 07:40:08 quay311omr systemd[1595]: Stopped Redis Podman Container for Quay.
Apr 11 07:40:08 quay311omr systemd[1595]: Starting Redis Podman Container for Quay...
Apr 11 07:40:08 quay311omr systemd[1595]: Started Redis Podman Container for Quay.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28074 (slirp4netns) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28093 (rootlessport) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28104 (exe) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28120 (conmon) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28143 (sleep) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: Starting Quay Container...
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28074 (slirp4netns) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28093 (rootlessport) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28104 (exe) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28120 (conmon) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28143 (sleep) in control group while starting unit. Ignoring.
Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Apr 11 07:40:08 quay311omr systemd[1595]: Started Quay Container.
Apr 11 07:40:08 quay311omr podman[34217]: Error: running container create option: no secret with name or id "redis_pass": no such secret
Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Main process exited, code=exited, status=125/n/a
Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Failed with result 'exit-code'.

        1. omr.logs
          861 kB

              Unassigned Unassigned
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: