When deploying hosted cluster it tries to reach registry.ci.openshift.org although internal mirror is configured correctly.

   [kni@ocp-edge77 ~]$ oc version
Client Version: 4.17.22
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.17.22
Kubernetes Version: v1.30.10
[kni@ocp-edge77 ~]$ ~/hypershift_working/hypershift/bin/hcp version
Client Version: openshift/hypershift: 6a6dad2e7a97462553515bd04e142060d7170068. Latest supported OCP: 4.17.0
Server Version: 6a6dad2e7a97462553515bd04e142060d7170068
Server Supports OCP Versions: 4.17, 4.16, 4.15, 4.14
 

    100%, blocking our CI for disconnected envir

    1.deploy a hub cluster, disconnected, ipv6, idms, with 3 masters 0 workers

(I used the cloned image after cleaning using thoes 2 jobs :
Clean : https://jenkins-csb-kniqe-auto.dno.corp.redhat.com/job/Infra/job/cleanup-all-terraform/10695/
Clone: https://jenkins-csb-kniqe-auto.dno.corp.redhat.com/job/Infra/job/factory-cluster-clone/738/
)

    2. deploy a hosted cluster(0 masters), with 2 worker nodes 

(I used this job to deploy :
https://jenkins-csb-kniqe-auto.dno.corp.redhat.com/job/CI/job/job-runner/4960/
)    

  the control plane is failing for puling image for unreachable network:

  - lastTransitionTime: "2025-04-09T17:19:04Z"
    message: 'failed to determine if release image multi-arch: failed to retrieve
      manifest registry.ci.openshift.org/ocp/release@sha256:65fbb79f801b621121793d347e4405d52edf114d7eea4f4b8c11115db57e5aa4:
      failed to create repository client for https://registry.ci.openshift.org: Get
      "https://registry.ci.openshift.org/v2/": dial tcp 54.161.197.18:443: connect:
      network is unreachable'

On the master node, the correct mirrors are on /etc/containers/registries.conf 

(core@master-0:/etc/containers/registries.conf)
And also podman pull is getting the image using the mirror 

when trying 'ocp image info' it is unreachable, unless it gets an explicit iscp file as a parameter.


[core@master-0 ~]$ oc image info registry.ci.openshift.org/ocp/release@sha256:65fbb79f801b621121793d347e4405d52edf114d7eea4f4b8c11115db57e5aa4
error: unable to read image registry.ci.openshift.org/ocp/release@sha256:65fbb79f801b621121793d347e4405d52edf114d7eea4f4b8c11115db57e5aa4: Get "https://registry.ci.openshift.org/v2/": dial tcp 54.161.197.18:443: connect: network is unreachableSo I've added --icsp-file I've added locally with the same mirrors, and it worked[core@master-0 ~]$ oc image info --icsp-file=icsp.yaml registry.ci.openshift.org/ocp/release@sha256:65fbb79f801b621121793d347e4405d52edf114d7eea4f4b8c11115db57e5aa4
Name:        registry.ci.openshift.org/ocp/release@sha256:65fbb79f801b621121793d347e4405d52edf114d7eea4f4b8c11115db57e5aa4
Media Type:  application/vnd.docker.distribution.manifest.v2+json
Created:     6d ago
Image Size:  168.5MB in 5 layers
Layers:      79.26MB sha256:25c75c34b2e2b68ba9245d9cddeb6b8a0887371ed30744064f85241a75704d87
             48.97MB sha256:a0a28f4b2d1284714d992bcd6e243462221181e8a4e9fbc407b7d6885aa67d89
             9.144MB sha256:55ebf42599e7ba5c5cd624c3188396e132d581bd7c0ab8360994574a477d311b
             29.47MB sha256:e5bde961eda32204412d83e0638ad389455a1bfbae44fcbef67343b9f5506ffc
             1.631MB sha256:c9676207ea4d91c9f778a8db33474d4ee2e51a9010018132f23020cb3ddcaeeb
OS:          linux
Arch:        amd64
Entrypoint:  /usr/bin/cluster-version-operator
Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             container=oci
             GODEBUG=x509ignoreCN=0,madvdontneed=1
             __doozer=merge
             BUILD_RELEASE=202503210032.p0.gb06c462.assembly.stream.el9
             BUILD_VERSION=v4.17.0
             OS_GIT_MAJOR=4
             OS_GIT_MINOR=17
             OS_GIT_PATCH=0
             OS_GIT_TREE_STATE=clean
             OS_GIT_VERSION=4.17.0-202503210032.p0.gb06c462.assembly.stream.el9-b06c462
             SOURCE_GIT_TREE_STATE=clean
             __doozer_group=openshift-4.17
             __doozer_key=cluster-version-operator
             __doozer_version=v4.17.0
             OS_GIT_COMMIT=b06c462
             SOURCE_DATE_EPOCH=1741774536
             SOURCE_GIT_COMMIT=b06c46276c7ac146925dc927202af97b66856a38
             SOURCE_GIT_TAG=v1.0.0-1252-gb06c4627
             SOURCE_GIT_URL=https://github.com/openshift/cluster-version-operator
             ART_BUILD_ENGINE=brew
             ART_BUILD_DEPS_METHOD=cachito
             ART_BUILD_NETWORK=internal-only
Labels:      io.openshift.release=4.17.24
             io.openshift.release.base-image-digest=sha256:6f425bd896fd835b9c07bdd3dd890e3859cb3221418eaaa76908ebf752209fc5

Expected results:

Image should be pulled from local registry as mirror is configured corectly
The control plane should be up and running    

post-mortem is here:
https://s3.upshift.redhat.com/DH-PROD-OCP-EDGE-QE-CI/ocp-spoke-assisted-operator-deploy/16711/index.html