-
Bug
-
Resolution: Cannot Reproduce
-
Undefined
-
None
-
4.12
-
Important
-
None
-
Proposed
-
False
-
-
Description of problem:
In a hub-spoke environment, that is connected to the internet. The hub fails to deploy the spoke due to errors in pulling images from quay.io.
Version-Release number of selected component (if applicable):
open shift 4.12 rc6 ACM 2.6.2-DOWNSTREAM-2022-10-14-21-53-49
How reproducible:
Here is the jenkins job that failed to deploy:
https://auto-jenkins-csb-kniqe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-far-edge-vran-deployment/2115/consoleFull
Console log:
https://drive.google.com/file/d/1SRrHfJ8bnp8_C_LefsR6twMDsAuIvhSs/view?usp=sharing
Steps to Reproduce:
1. Deploy hub cluster 2. Deploy SNO with jenkins job above
Actual results:
Ansible timed-out at TASK [deploy-gitops-du : wait for installation to finish]
Expected results:
Deploy used to complete fine on this setup a week ago
Additional info:
I ssh to the spoke and see this in the journal:
Jan 02 11:47:02 ocp-edge90.lab.eng.tlv2.redhat.com next_step_runne[3780]: time="02-01-2023 11:47:02" level=error msg="Step execution failed (exit code 2): <container-image-availability-844caec8>, command: <container_image_availability>, args: <[{\"images\":[\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-release:4.11.4-x86_64\",\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev@sha256:21eb9adf82d7949794817f7ecfdd13577c63b7ceefe86ab3671929a179dd883b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb\",\"registry.redhat.io/multicluster-engine/assisted-installer-rhel8@sha256:05b30083201dcf9ea06d502586649c699af83034afa0f9967fb5440666f5f81b\"],\"timeout\":960}]>. Output:\nstdout:\n{\"images\":[{\"name\":\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-release:4.11.4-x86_64\",\"result\":\"success\"},{\"name\":\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev@sha256:21eb9adf82d7949794817f7ecfdd13577c63b7ceefe86ab3671929a179dd883b\",\"result\":\"success\"},{\"name\":\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb\",\"result\":\"failure\"},{\"name\":\"registry.redhat.io/multicluster-engine/assisted-installer-rhel8@sha256:05b30083201dcf9ea06d502586649c699af83034afa0f9967fb5440666f5f81b\",\"result\":\"success\"}]}\n\nstderr:\n\n" file="step_processor.go:108" request_id=5598cae7-aad0-4053-bd1d-768e1a6061c9
I see the spoke failed to pull the image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb
due to: unauthorized: access to the requested resource is not authorized
So in the command line of ocp-edge90 I issue:
[core@ocp-edge90 ~]$ podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb
And get:
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb...
Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: (Mirrors also failed: [registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: reading manifest sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb in registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: reading manifest sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized
This pull error appears 285 time in the journal for the same image.
- blocks
-
-
- Closed
-
