Customer is trying to mirror 4.19.10 in a disconnected cluster but the process is failing due to a manifest unknown. The yaml configs were already set: (bottom of the comment)
1 - The error log
[evuser@deployment-controller cluster-resources]$ /usr/local/bin/oc adm release extract -a /home/evuser/registry_auth.json --icsp-file=icsp-oc-mirror.yaml --insecure=true --command=openshift-install --to=/tmp mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64 -v=10
Flag --icsp-file has been deprecated, support for it will be removed in a future release. Use --idms-file instead.
warning: --idms-file(and --icsp-file) only applies to images referenced by digest and will be ignored for tags
I1014 16:57:04.496502 63996 round_trippers.go:473] curl -v -XGET -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/'
I1014 16:57:04.496734 63996 round_trippers.go:502] HTTP Trace: DNS Lookup for mirror-registry.ems.energyvault.com resolved to [
]
I1014 16:57:04.497000 63996 round_trippers.go:517] HTTP Trace: Dial to tcp:10.0.3.50:5000 succeed
I1014 16:57:04.505336 63996 round_trippers.go:560] GET https://mirror-registry.ems.energyvault.com:5000/v2/ 401 Unauthorized in 8 milliseconds
I1014 16:57:04.505360 63996 round_trippers.go:577] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 7 ms ServerProcessing 0 ms Duration 8 ms
I1014 16:57:04.505365 63996 round_trippers.go:584] Response Headers:
I1014 16:57:04.505371 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
I1014 16:57:04.505375 63996 round_trippers.go:587] Content-Type: application/json
I1014 16:57:04.505378 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
I1014 16:57:04.505383 63996 round_trippers.go:587] Www-Authenticate: Basic realm="Registry Realm"
I1014 16:57:04.505388 63996 round_trippers.go:587] Content-Length: 87
I1014 16:57:04.506040 63996 round_trippers.go:473] curl -v -XHEAD -H "Authorization: Basic <masked>" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64'
I1014 16:57:04.697360 63996 round_trippers.go:560] HEAD https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64 404 Not Found in 191 milliseconds
I1014 16:57:04.697389 63996 round_trippers.go:577] HTTP Statistics: GetConnection 0 ms ServerProcessing 191 ms Duration 191 ms
I1014 16:57:04.697400 63996 round_trippers.go:584] Response Headers:
I1014 16:57:04.697410 63996 round_trippers.go:587] Content-Type: application/json
I1014 16:57:04.697425 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
I1014 16:57:04.697435 63996 round_trippers.go:587] Content-Length: 209
I1014 16:57:04.697443 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
I1014 16:57:04.697501 63996 round_trippers.go:473] curl -v -XGET -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" -H "Accept: application/vnd.oci.image.index.v1+json" -H "Accept: application/vnd.oci.image.manifest.v1+json" -H "Accept: application/vnd.docker.distribution.manifest.v1+prettyjws" -H "Accept: application/json" -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Basic <masked>" -H "User-Agent: oc/4.19.0 (linux/amd64) kubernetes/298429b" 'https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64'
I1014 16:57:04.888065 63996 round_trippers.go:560] GET https://mirror-registry.ems.energyvault.com:5000/v2/openshift/release-images/manifests/4.19.10-x86_64 404 Not Found in 190 milliseconds
I1014 16:57:04.888093 63996 round_trippers.go:577] HTTP Statistics: GetConnection 0 ms ServerProcessing 190 ms Duration 190 ms
I1014 16:57:04.888103 63996 round_trippers.go:584] Response Headers:
I1014 16:57:04.888114 63996 round_trippers.go:587] Content-Length: 209
I1014 16:57:04.888123 63996 round_trippers.go:587] Date: Tue, 14 Oct 2025 16:57:04 GMT
I1014 16:57:04.888130 63996 round_trippers.go:587] Content-Type: application/json
I1014 16:57:04.888139 63996 round_trippers.go:587] Docker-Distribution-Api-Version: registry/2.0
I1014 16:57:04.888272 63996 workqueue.go:143] about to send work queue error: image "mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64" not found: manifest unknown: manifest unknown
error: image "mirror-registry.ems.energyvault.com:5000/openshift/release-images:4.19.10-x86_64" not found: manifest unknown: manifest unknown
~~~"
2 - Tested the openshift-install
[evuser@deployment-controller ~]$ openshift-install agent create image --dir /data/openshift-install --log-level debug
DEBUG OpenShift Installer 4.19.14
DEBUG Built from commit 7e30b7d6b421087ee6b6aaa639e40392c22ce52b
DEBUG Fetching Agent Installer ISO...
DEBUG Loading Agent Installer ISO...
(...)
DEBUG Generating Agent Installer Artifacts...
DEBUG Using mirror configuration
DEBUG Fetching image from OCP release ([oc adm release info --image-for=agent-installer-utils --filter-by-os=linux/amd64 --insecure=true --icsp-file=/tmp/icsp-file923530913 quay.io/openshift-release-dev/ocp-release@sha256:f8e21e76897b3f9b8a76a07b5a9426ba8def9b2e56b18d8b40ad65931b8bbf78])
DEBUG extracting /usr/bin/agent-tui to /home/evuser/.cache/agent/files_cache, [oc image extract --path=/usr/bin/agent-tui:/home/evuser/.cache/agent/files_cache --filter-by-os=linux/amd64 --insecure=true --confirm --icsp-file=/tmp/icsp-file1567689105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5]
ERROR failed to write asset (Agent Installer ISO) to disk: cannot generate ISO image due to configuration errors
FATAL failed to fetch Agent Installer ISO: failed to fetch dependency of "Agent Installer ISO": failed to generate asset "Agent Installer Artifacts": command '[oc image extract --path=/usr/bin/agent-tui:/home/evuser/.cache/agent/files_cache --filter-by-os=linux/amd64 --insecure=true --confirm --icsp-file=/tmp/icsp-file1567689105 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5 --registry-config=/tmp/registry-config330406590]' exited with non-zero exit code 1:
FATAL Flag --icsp-file has been deprecated, support for it will be removed in a future release. Use --idms-file instead.
FATAL error: unable to read image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d94ae4fefca3ae786a1433394a93e0af7ebb320dca48278c17b0f553899ec9f5: unauthorized: access to the requested resource is not authorized
FATAL
3 - I tested the crictl pull in the Resource Hub cluster and it is failing. Signature Digest worked properly:
3.1 - From the oc-mirror debug: (could not find destination entry).
2025/10/10 14:19:34 [1;92m [DEBUG] [0m : [ReleaseImageCollector] image manifest digest 034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9
2025/10/10 14:19:34 [1;92m [DEBUG] [0m : [ReleaseImageCollector] config digest
2025/10/10 14:19:34 [1;92m [DEBUG] [0m : extract directory exists (nop)
3.2 - Tested the pull manually:
crictl pull quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9
E1014 18:34:28.200214 2053318 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = initializing source docker://quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9: reading manifest sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9 in quay.io/openshift-release-dev/ocp-release: manifest unknown" image="quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9"
FATA[0001] pulling image: initializing source docker://quay.io/openshift-release-dev/ocp-release@sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9: reading manifest sha256:034270e0ac799f9f9c939ff24493771c09a8cce219dbb654b7a3629fad4ab0c9 in quay.io/openshift-release-dev/ocp-release: manifest unknown
4- The yaml configs
apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
annotations:
createdAt: Wednesday, 08-Oct-25 22:33:03 UTC
createdBy: oc-mirror v2
oc-mirror_version: 4.19.0-202509230411.p2.g0a5c2ac.assembly.stream.el9-0a5c2ac
name: itms-release-0
spec:
imageTagMirrors:
- mirrors:
- mirror-registry.ems.energyvault.com:5000/openshift/release-images
source: quay.io/openshift-release-dev/ocp-release - mirrors:
- mirror-registry.ems.energyvault.com:5000/openshift/release-images
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
status: {}
~~~
and also added your ICSP:
~~~
[evuser@deployment-controller cluster-resources]$ cat icsp-oc-mirror.yaml
imageContentSources:
- mirrors:
- mirror-registry.ems.energyvault.com:5000/openshift/release-images
source: quay.io/openshift-release-dev/ocp-release - mirrors:
- mirror-registry.ems.energyvault.com:5000/openshift/release-images
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev