-
Bug
-
Resolution: Done-Errata
-
Blocker
-
omr-v1.3.11
-
False
-
None
-
False
-
-
Description:
This is an issue found in OMR 1.3.11 upgrade testing, after triggered OMR 1.3.11 upgrade from OMR 1.3.10, the OMR upgrade was failed at step "TASK [mirror_appliance : Wait for Quay]", checked the logs, get error message with "Error: running container create option: no secret with name or id "pgdb_pass": no such secret", "Error: running container create option: no secret with name or id "redis_pass": no such secret", pls review attached logs omr.logs
OMR: openshift-mirror-registry-rhel8:v1.3.11-5
OMR Upgrade Logs:
./mirror-registry --version __ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your Containersmirror-registry version v1.3.11 ./mirror-registry upgrade -H quayomr.qe.gcp.devcluster.openshift.com __ __ / \ / \ ______ _ _ __ __ __ / /\ / /\ \ / __ \ | | | | / \ \ \ / / / / / / \ \ | | | | | | | | / /\ \ \ / \ \ \ \ / / | |__| | | |__| | / ____ \ | | \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| \__/ \__/ \ \__ \___\ by Red Hat Build, Store, and Distribute your ContainersINFO[2024-04-11 07:22:18] Upgrade has begun INFO[2024-04-11 07:22:18] Found execution environment at /home/lizhang/execution-environment.tar INFO[2024-04-11 07:22:18] Loading execution environment from execution-environment.tar INFO[2024-04-11 07:22:22] Found SSH key at /home/lizhang/.ssh/quay_installer INFO[2024-04-11 07:22:22] Attempting to set SELinux rules on /home/lizhang/.ssh/quay_installer INFO[2024-04-11 07:22:22] Found image archive at /home/lizhang/image-archive.tar INFO[2024-04-11 07:22:22] Attempting to set SELinux rules on image archive INFO[2024-04-11 07:22:22] Running upgrade playbook. This may take some time. To see playbook output run the installer with -v (verbose) flag. PLAY [Upgrade Mirror Appliance] *******************************************************************************************************************************TASK [Gathering Facts] **************************************************************************************************************************************** ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand variables] ******************************************************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/expand-vars.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Expand pg_storage] ******************************************************************************************************************* changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand quay_root] ******************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Expand quay_storage] ***************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Set expanded variables] ************************************************************************************************************** ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Install Dependencies] **************************************************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/install-deps.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Create user service directory] ******************************************************************************************************* ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Set SELinux Rules] ******************************************************************************************************************* included: /runner/project/roles/mirror_appliance/tasks/set-selinux-rules.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Set container_manage_cgroup flag on and keep it persistent across reboots] *********************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Autodetect root or rootless from previous install] *********************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/migrate-pre-1_3_0.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Check for old quay-storage] ********************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Stop Quay service] ******************************************************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Create temporary data-only container for quay-storage] ******************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy data to quay-storage volume] **************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete temporary container] ********************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete /etc/quay-install/quay-storage] *********************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check for old pg-storage] ************************************************************************************************************ skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Stop Postgres service] *************************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Create temporary data-only container for pg-storage] ********************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy data to pg-storage volume] ****************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Fix permissions] ********************************************************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete temporary container] ********************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Delete /etc/quay-install/pg-data] **************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check for old Quay root /etc/quay-install] ******************************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Migrate Quay root] ******************************************************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Autodetect Image Archive] ************************************************************************************************************ included: /runner/project/roles/mirror_appliance/tasks/autodetect-image-archive.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Checking for Image Archive] ********************************************************************************************************** ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com -> localhost]TASK [mirror_appliance : Create install directory for image-archive.tar dest] ********************************************************************************* ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Copy Images if /runner/image-archive.tar exists] ************************************************************************************* changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Unpack Images if /runner/image-archive.tar exists] *********************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Redis if redis.tar exists] *************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Quay if quay.tar exists] ***************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Loading Postgres if postgres.tar exists] ********************************************************************************************* changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Quay Pod Service] ************************************************************************************************************ included: /runner/project/roles/mirror_appliance/tasks/upgrade-pod-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Quay Pod systemd service file] ************************************************************************************************** ok: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if pod pause image is loaded] ************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Infra image] ******************************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Quay Pod service] ************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Postgres Service] ************************************************************************************************************ included: /runner/project/roles/mirror_appliance/tasks/upgrade-postgres-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Postgres systemd service file] ************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Postgres image is loaded] *************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Postgres image] ***************************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Postgres service] ************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Redis Service] *************************************************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/upgrade-redis-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Redis systemd service file] ***************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Redis image is loaded] ****************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Redis image] ******************************************************************************************************************** skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Redis service] ***************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Upgrade Quay Service] **************************************************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/upgrade-quay-service.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Copy Quay systemd service file] ****************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Check if Quay image is loaded] ******************************************************************************************************* changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Pull Quay image] ********************************************************************************************************************* skipping: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Start Quay service] ****************************************************************************************************************** changed: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]TASK [mirror_appliance : Wait for Quay] *********************************************************************************************************************** included: /runner/project/roles/mirror_appliance/tasks/wait-for-quay.yaml for lizhang@quayomr.qe.gcp.devcluster.openshift.comTASK [mirror_appliance : Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance] ************ FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (10 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (9 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (8 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (7 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (6 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (5 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (4 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (3 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (2 retries left). FAILED - RETRYING: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: Waiting up to 3 minutes for Quay to become alive at https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance (1 retries left). fatal: [lizhang@quayomr.qe.gcp.devcluster.openshift.com]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 104] Connection reset by peer>", "redirected": false, "status": -1, "url": "https://quayomr.qe.gcp.devcluster.openshift.com:8443/health/instance"}PLAY RECAP **************************************************************************************************************************************************** lizhang@quayomr.qe.gcp.devcluster.openshift.com : ok=35 changed=19 unreachable=0 failed=1 skipped=20 rescued=0 ignored=0 ERRO[2024-04-11 07:31:10] An error occurred: exit status 2
OMR upgrade errors:
Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Service RestartSec=30s expired, scheduling restart. Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Scheduled restart job, restart counter is at 29. Apr 11 07:40:06 quay311omr systemd[1595]: Stopped PostgreSQL Podman Container for Quay. Apr 11 07:40:06 quay311omr systemd[1595]: Starting PostgreSQL Podman Container for Quay... Apr 11 07:40:06 quay311omr systemd[1595]: Started PostgreSQL Podman Container for Quay. Apr 11 07:40:06 quay311omr podman[34198]: Error: running container create option: no secret with name or id "pgdb_pass": no such secret Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Main process exited, code=exited, status=125/n/a Apr 11 07:40:06 quay311omr systemd[1595]: quay-postgres.service: Failed with result 'exit-code'. Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Service RestartSec=30s expired, scheduling restart. Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Scheduled restart job, restart counter is at 29. Apr 11 07:40:08 quay311omr systemd[1595]: Stopped Redis Podman Container for Quay. Apr 11 07:40:08 quay311omr systemd[1595]: Starting Redis Podman Container for Quay... Apr 11 07:40:08 quay311omr systemd[1595]: Started Redis Podman Container for Quay. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28074 (slirp4netns) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28093 (rootlessport) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28104 (exe) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28120 (conmon) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28143 (sleep) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: Starting Quay Container... Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28074 (slirp4netns) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28093 (rootlessport) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28104 (exe) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28120 (conmon) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: quay-app.service: Found left-over process 28143 (sleep) in control group while starting unit. Ignoring. Apr 11 07:40:08 quay311omr systemd[1595]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Apr 11 07:40:08 quay311omr systemd[1595]: Started Quay Container. Apr 11 07:40:08 quay311omr podman[34217]: Error: running container create option: no secret with name or id "redis_pass": no such secret Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Main process exited, code=exited, status=125/n/a Apr 11 07:40:08 quay311omr systemd[1595]: quay-redis.service: Failed with result 'exit-code'.
- links to
-
RHBA-2024:1758
mirror registry for Red Hat OpenShift 1.3.11 container
- mentioned on
