Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5256

ZTP deploy of SNO fails due to failing to pull images in a connected enviromant

XMLWordPrintable

    • Important
    • Proposed
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      In a hub-spoke environment, that is connected to the internet. The hub fails to deploy the spoke due to errors in pulling images from quay.io.

      Version-Release number of selected component (if applicable):

      open shift 4.12 rc6
ACM 2.6.2-DOWNSTREAM-2022-10-14-21-53-49

      How reproducible:

      Here is the jenkins job that failed to deploy:
      https://auto-jenkins-csb-kniqe.apps.ocp-c1.prod.psi.redhat.com/job/ocp-far-edge-vran-deployment/2115/consoleFull
      Console log:
      https://drive.google.com/file/d/1SRrHfJ8bnp8_C_LefsR6twMDsAuIvhSs/view?usp=sharing

      Steps to Reproduce:

      1. Deploy hub cluster
2. Deploy SNO with jenkins job above

      Actual results:

      Ansible timed-out at TASK [deploy-gitops-du : wait for installation to finish]

      Expected results:

      Deploy used to complete fine on this setup a week ago

      Additional info:

      I ssh to the spoke and see this in the journal:

      Jan 02 11:47:02 ocp-edge90.lab.eng.tlv2.redhat.com next_step_runne[3780]: time="02-01-2023 11:47:02" level=error msg="Step execution failed (exit code 2): <container-image-availability-844caec8>, command: <container_image_availability>, args: <[{\"images\":[\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-release:4.11.4-x86_64\",\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev@sha256:21eb9adf82d7949794817f7ecfdd13577c63b7ceefe86ab3671929a179dd883b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb\",\"registry.redhat.io/multicluster-engine/assisted-installer-rhel8@sha256:05b30083201dcf9ea06d502586649c699af83034afa0f9967fb5440666f5f81b\"],\"timeout\":960}]>. Output:\nstdout:\n{\"images\":[{\"name\":\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-release:4.11.4-x86_64\",\"result\":\"success\"},{\"name\":\"registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev@sha256:21eb9adf82d7949794817f7ecfdd13577c63b7ceefe86ab3671929a179dd883b\",\"result\":\"success\"},{\"name\":\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb\",\"result\":\"failure\"},{\"name\":\"registry.redhat.io/multicluster-engine/assisted-installer-rhel8@sha256:05b30083201dcf9ea06d502586649c699af83034afa0f9967fb5440666f5f81b\",\"result\":\"success\"}]}\n\nstderr:\n\n" file="step_processor.go:108" request_id=5598cae7-aad0-4053-bd1d-768e1a6061c9

      I see the spoke failed to pull the image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb

      due to: unauthorized: access to the requested resource is not authorized 

       

      So in the command line of ocp-edge90 I issue:

      [core@ocp-edge90 ~]$ podman pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb

      And get:

      Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb...
Error: initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: (Mirrors also failed: [registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: reading manifest sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb in registry.kni-qe-19.lab.eng.tlv2.redhat.com:5000/openshift-release-dev/ocp-v4.0-art-dev: manifest unknown: manifest unknown]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb: reading manifest sha256:837527bb57d6b1f1b81cc2c0b405736d6c228cbf134e445beaa87766ca2ffedb in quay.io/openshift-release-dev/ocp-v4.0-art-dev: unauthorized: access to the requested resource is not authorized

      This pull error appears 285 time in the journal for the same image.

            kenyis Ken Young
            rhn-support-nwaizer Niv Gal Waizer (Inactive)
            Yang Liu Yang Liu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: