-
Bug
-
Resolution: Done
-
Critical
-
None
-
4.16, 4.17
Description of problem:
When running oc-mirror V2 (either 4.16 or 4.17 has been tested) on a RHEL 9 FIPS enabled and STIG Security profile enforced system, oc-mirror fails due to a hard coded PGP key in oc-mirror V2.
Version-Release number of selected component (if applicable):
At least 4.16-4.17
How reproducible:
Very reproducible
Steps to Reproduce:
- Install latest oc-mirror
- Create a cluster-images.yml file
ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.16
minVersion: 4.16.18
maxVersion: 4.16.24
shortestPath: true
3. run oc-mirror with the following flags:
[cnovak@localhost ocp4-disconnected-config]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v2 2024/12/18 14:40:01 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/12/18 14:40:01 [INFO] : 👋 Hello, welcome to oc-mirror 2024/12/18 14:40:01 [INFO] : ⚙️ setting up the environment for you... 2024/12/18 14:40:01 [INFO] : 🔀 workflow mode: mirrorToDisk 2024/12/18 14:40:01 [INFO] : 🕵️ going to discover the necessary images... 2024/12/18 14:40:01 [INFO] : 🔍 collecting release images... 2024/12/18 14:40:02 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure 2024/12/18 14:40:02 [ERROR] : generate release signatures: error list invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 2024/12/18 14:40:02 [INFO] : 🔍 collecting operator images... 2024/12/18 14:40:02 [INFO] : 🔍 collecting additional images... 2024/12/18 14:40:02 [INFO] : 🚀 Start copying the images... 2024/12/18 14:40:02 [INFO] : images to copy 0 2024/12/18 14:40:02 [INFO] : === Results === 2024/12/18 14:40:02 [INFO] : 📦 Preparing the tarball archive... 2024/12/18 14:40:02 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2024/12/18 14:40:02 [ERROR] : unable to add cache repositories to the archive : lstat /home/cnovak/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
Expected results/immediate workaround:
[cnovak@localhost ~]$ curl -s https://raw.githubusercontent.com/openshift/cluster-update-keys/d44fca585d081a72cb2c67734556a27bbfc9470e/manifests.rhel/0000_90_cluster-update-keys_configmap.yaml | sed -n '/openshift[.]io/d;s/Comment:.*//;s/^ //p' > /tmp/pgpkey [cnovak@localhost ~]$ export OCP_SIGNATURE_VERIFICATION_PK=/tmp/pgpkey [cnovak@localhost ~]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v22024/12/19 08:54:42 [WARN] : ⚠️ --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2024/12/19 08:54:42 [INFO] : 👋 Hello, welcome to oc-mirror 2024/12/19 08:54:42 [INFO] : ⚙️ setting up the environment for you... 2024/12/19 08:54:42 [INFO] : 🔀 workflow mode: mirrorToDisk 2024/12/19 08:54:42 [INFO] : 🕵️ going to discover the necessary images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting release images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting operator images... 2024/12/19 08:54:42 [INFO] : 🔍 collecting additional images... 2024/12/19 08:54:42 [INFO] : 🚀 Start copying the images... 2024/12/19 08:54:42 [INFO] : images to copy 382 ⠸ 1/382 : (7s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb ✓ 2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace ✓ 3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 ✓ 4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 ⠋ 1/382 : (19s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb ✓ 2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace ✓ 3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 ✓ 4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 ✓ 5/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e54fc21197c341fe257d2f2f2ad14b578483c4450474dc2cf876a885f11e745 ✓ 6/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c934b4d95545e29f9cb7586964fd43cdb7b8533619961aaa932fe2923ab40db ✓ 7/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:851ba9ac5219a9f11e927200715e666ae515590cd9cc6dde9631070afb66b5d7 ✓ 8/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f614ef855220f2381217c31b8cb94c05ef20edf3ca23b5efa0be1b957cdde3a4
Additional info:
The reason this is a critical issue, is Red Hat has a relatively large footprint within the DoD/U.S Government space, and anyone who is working in a disconnected environment, with a STIG Policy enforced on a RHEL 9 machine, will run into this problem.
Additionally, below is output from oc-mirror version
[cnovak@localhost ~]$ oc-mirror version
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.17.0-202411251634.p0.g07714b7.assembly.stream.el9-07714b7", GitCommit:"07714b7c836ec3ad1b776f25b44c3b2c2f083aa2", GitTreeState:"clean", BuildDate:"2024-11-26T08:28:42Z", GoVersion:"go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
- blocks
-
-
- Closed
-
- causes
-
-
- Closed
-
- is cloned by
-
-
- Closed
-
- is triggered by
-
-
- Closed
-
- is triggering
-
CLID-356 Corrective Measure for OCPBUGS-47453: oc-mirror V2 fails on FIPS enabled and STIG compliant RHEL 9 system
-
- To Do
-
- relates to
-
-
- Closed
-
- links to
-
RHEA-2024:11038
OpenShift Container Platform 4.19.z bug fix update
(1 relates to, 3 links to)
[OCPBUGS-47453] oc-mirror V2 fails on FIPS enabled and STIG compliant RHEL 9 system
Rama Kasturi Narra
added a comment - Reproduced the issue on 4.19 cluster with 4.19 ec0 oc-mirror bits and could see the error as described above.
[root@ip-10-0-20-114 tmp]# ./oc-mirror version
W0116 10:44:36.791903 25840 mirror.go:102]
⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info
[root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file://images --v2 2025/01/16 10:44:21 [INFO] : 👋 Hello, welcome to oc-mirror 2025/01/16 10:44:21 [INFO] : ⚙️ setting up the environment for you... 2025/01/16 10:44:21 [INFO] : 🔀 workflow mode: mirrorToDisk 2025/01/16 10:44:21 [INFO] : 🕵️ going to discover the necessary images... 2025/01/16 10:44:21 [INFO] : 🔍 collecting release images... 2025/01/16 10:44:22 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure 2025/01/16 10:44:22 [ERROR] : [GenerateReleaseSignatures] invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 2025/01/16 10:44:22 [INFO] : 🔍 collecting operator images... 2025/01/16 10:44:22 [INFO] : 🔍 collecting additional images... 2025/01/16 10:44:22 [INFO] : 🔍 collecting helm images... 2025/01/16 10:44:22 [INFO] : 🔂 rebuilding catalogs 2025/01/16 10:44:22 [INFO] : 🚀 Start copying the images... 2025/01/16 10:44:22 [INFO] : images to copy 0 2025/01/16 10:44:22 [INFO] : === Results === 2025/01/16 10:44:22 [INFO] : 📦 Preparing the tarball archive... 2025/01/16 10:44:22 [INFO] : 👋 Goodbye, thank you for using oc-mirror 2025/01/16 10:44:22 [ERROR] : unable to add cache repositories to the archive : lstat /root/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
Extracted oc-mirror bits from latest nightly and i see the issue has been fixed:
[root@ip-10-0-20-114 tmp]# ./oc-mirror version
W0116 10:50:00.701661 28489 mirror.go:102]
⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info
[root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file://images --v2 --dry-run 2025/01/16 10:50:31 [INFO] : 👋 Hello, welcome to oc-mirror 2025/01/16 10:50:31 [INFO] : ⚙️ setting up the environment for you... 2025/01/16 10:50:31 [INFO] : 🔀 workflow mode: mirrorToDisk 2025/01/16 10:50:31 [INFO] : 🕵️ going to discover the necessary images... 2025/01/16 10:50:31 [INFO] : 🔍 collecting release images... 2025/01/16 10:50:41 [INFO] : 🔍 collecting operator images... 2025/01/16 10:50:41 [INFO] : 🔍 collecting additional images... 2025/01/16 10:50:41 [INFO] : 🔍 collecting helm images... 2025/01/16 10:50:42 [WARN] : ⚠️ 384/384 images necessary for mirroring are not available in the cache. 2025/01/16 10:50:42 [WARN] : List of missing images in : images/working-dir/dry-run/missing.txt. please re-run the mirror to disk process 2025/01/16 10:50:42 [INFO] : 📄 list of all images for mirroring in : images/working-dir/dry-run/mapping.txt 2025/01/16 10:50:42 [INFO] : mirror time : 10.878046203s 2025/01/16 10:50:42 [INFO] : 👋 Goodbye, thank you for using oc-mirror [root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file://images --v2 --authfile auth 2025/01/16 10:59:04 [INFO] : 👋 Hello, welcome to oc-mirror 2025/01/16 10:59:04 [INFO] : ⚙️ setting up the environment for you... 2025/01/16 10:59:04 [INFO] : 🔀 workflow mode: mirrorToDisk 2025/01/16 10:59:04 [INFO] : 🕵️ going to discover the necessary images... 2025/01/16 10:59:04 [INFO] : 🔍 collecting release images... 2025/01/16 10:59:04 [INFO] : 🔍 collecting operator images... 2025/01/16 10:59:04 [INFO] : 🔍 collecting additional images... 2025/01/16 10:59:04 [INFO] : 🔍 collecting helm images... 2025/01/16 10:59:04 [INFO] : 🔂 rebuilding catalogs 2025/01/16 10:59:04 [INFO] : 🚀 Start copying the images... 2025/01/16 10:59:04 [INFO] : images to copy 384 ✓ 1/384 : (2s) quay.io/openshift-release-dev/ocp-release:4.16.18-x86_64 ➡️ cache ✓ 2/384 : (0s) quay.io/openshift-release-dev/ocp-release:4.16.24-x86_64 ➡️ cache ✓ 3/384 : (2s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:004f71e845f34bf2de0933ca3930262f4ee5054f09bfd6008e2486623e54d87e ➡️ cache ✓ 384/384 : (4s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fefbbd3732f8c3e1674d0db26350821d6d3fe2ea3c146de2272ebacec48b4406 ➡️ cache 2025/01/16 11:13:06 [INFO] : === Results === 2025/01/16 11:13:06 [INFO] : ✅ 384 / 384 release images mirrored successfully 2025/01/16 11:13:06 [INFO] : 📦 Preparing the tarball archive... 2025/01/16 11:22:45 [INFO] : mirror time : 23m41.315768447s 2025/01/16 11:22:45 [INFO] : 👋 Goodbye, thank you for using oc-mirror
Based on the above moving bug to verified state.
Rama Kasturi Narra
added a comment - Reproduced the issue on 4.19 cluster with 4.19 ec0 oc-mirror bits and could see the error as described above.
[root@ip-10-0-20-114 tmp] # ./oc-mirror version
W0116 10:44:36.791903 25840 mirror.go:102]
⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info
{Major:"", Minor:"", GitVersion:"4.19.0-202412051538.p0.g33e51c5.assembly.stream.el9-33e51c5", GitCommit:"33e51c5ce34382425608af77a0642376df2e15c5", GitTreeState:"clean", BuildDate:"2024-12-05T18:47:25Z", GoVersion:"go1.23.2 (Red Hat 1.23.2-1.el9) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
[root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file: //images --v2
2025/01/16 10:44:21 [INFO] : 👋 Hello, welcome to oc-mirror
2025/01/16 10:44:21 [INFO] : ⚙️ setting up the environment for you...
2025/01/16 10:44:21 [INFO] : 🔀 workflow mode: mirrorToDisk
2025/01/16 10:44:21 [INFO] : 🕵️ going to discover the necessary images...
2025/01/16 10:44:21 [INFO] : 🔍 collecting release images...
2025/01/16 10:44:22 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure
2025/01/16 10:44:22 [ERROR] : [GenerateReleaseSignatures] invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7
2025/01/16 10:44:22 [INFO] : 🔍 collecting operator images...
2025/01/16 10:44:22 [INFO] : 🔍 collecting additional images...
2025/01/16 10:44:22 [INFO] : 🔍 collecting helm images...
2025/01/16 10:44:22 [INFO] : 🔂 rebuilding catalogs
2025/01/16 10:44:22 [INFO] : 🚀 Start copying the images...
2025/01/16 10:44:22 [INFO] : images to copy 0
2025/01/16 10:44:22 [INFO] : === Results ===
2025/01/16 10:44:22 [INFO] : 📦 Preparing the tarball archive...
2025/01/16 10:44:22 [INFO] : 👋 Goodbye, thank you for using oc-mirror
2025/01/16 10:44:22 [ERROR] : unable to add cache repositories to the archive : lstat /root/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
Extracted oc-mirror bits from latest nightly and i see the issue has been fixed:
[root@ip-10-0-20-114 tmp] # ./oc-mirror version
W0116 10:50:00.701661 28489 mirror.go:102]
⚠️ oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2
WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
Client Version: version.Info
{Major:"", Minor:"", GitVersion:"4.19.0-202501131109.p0.g3617ded.assembly.stream.el9-3617ded", GitCommit:"3617dedcbdf20b1792764d4c2381e8490db32140", GitTreeState:"clean", BuildDate:"2025-01-13T11:52:29Z", GoVersion:"go1.23.2 (Red Hat 1.23.2-1.el9) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
[root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file: //images --v2 --dry-run
2025/01/16 10:50:31 [INFO] : 👋 Hello, welcome to oc-mirror
2025/01/16 10:50:31 [INFO] : ⚙️ setting up the environment for you...
2025/01/16 10:50:31 [INFO] : 🔀 workflow mode: mirrorToDisk
2025/01/16 10:50:31 [INFO] : 🕵️ going to discover the necessary images...
2025/01/16 10:50:31 [INFO] : 🔍 collecting release images...
2025/01/16 10:50:41 [INFO] : 🔍 collecting operator images...
2025/01/16 10:50:41 [INFO] : 🔍 collecting additional images...
2025/01/16 10:50:41 [INFO] : 🔍 collecting helm images...
2025/01/16 10:50:42 [WARN] : ⚠️ 384/384 images necessary for mirroring are not available in the cache.
2025/01/16 10:50:42 [WARN] : List of missing images in : images/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
2025/01/16 10:50:42 [INFO] : 📄 list of all images for mirroring in : images/working-dir/dry-run/mapping.txt
2025/01/16 10:50:42 [INFO] : mirror time : 10.878046203s
2025/01/16 10:50:42 [INFO] : 👋 Goodbye, thank you for using oc-mirror
[root@ip-10-0-20-114 tmp]# ./oc-mirror -c isc.yaml file: //images --v2 --authfile auth
2025/01/16 10:59:04 [INFO] : 👋 Hello, welcome to oc-mirror
2025/01/16 10:59:04 [INFO] : ⚙️ setting up the environment for you...
2025/01/16 10:59:04 [INFO] : 🔀 workflow mode: mirrorToDisk
2025/01/16 10:59:04 [INFO] : 🕵️ going to discover the necessary images...
2025/01/16 10:59:04 [INFO] : 🔍 collecting release images...
2025/01/16 10:59:04 [INFO] : 🔍 collecting operator images...
2025/01/16 10:59:04 [INFO] : 🔍 collecting additional images...
2025/01/16 10:59:04 [INFO] : 🔍 collecting helm images...
2025/01/16 10:59:04 [INFO] : 🔂 rebuilding catalogs
2025/01/16 10:59:04 [INFO] : 🚀 Start copying the images...
2025/01/16 10:59:04 [INFO] : images to copy 384
✓ 1/384 : (2s) quay.io/openshift-release-dev/ocp-release:4.16.18-x86_64 ➡️ cache
✓ 2/384 : (0s) quay.io/openshift-release-dev/ocp-release:4.16.24-x86_64 ➡️ cache
✓ 3/384 : (2s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:004f71e845f34bf2de0933ca3930262f4ee5054f09bfd6008e2486623e54d87e ➡️ cache
✓ 384/384 : (4s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fefbbd3732f8c3e1674d0db26350821d6d3fe2ea3c146de2272ebacec48b4406 ➡️ cache
2025/01/16 11:13:06 [INFO] : === Results ===
2025/01/16 11:13:06 [INFO] : ✅ 384 / 384 release images mirrored successfully
2025/01/16 11:13:06 [INFO] : 📦 Preparing the tarball archive...
2025/01/16 11:22:45 [INFO] : mirror time : 23m41.315768447s
2025/01/16 11:22:45 [INFO] : 👋 Goodbye, thank you for using oc-mirror
Based on the above moving bug to verified state. 
OpenShift Jira Bot
added a comment - Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi skhoury@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. Rama Kasturi Narra
added a comment - skhoury@redhat.com Have updated bug with all the results that i have got, could you please check and let me know if they look okay and if any other validation is required so that i can add qe-approved to the PR ? thanks !!
Rama Kasturi Narra
added a comment - skhoury@redhat.com Have updated bug with all the results that i have got, could you please check and let me know if they look okay and if any other validation is required so that i can add qe-approved to the PR ? thanks !! Rama Kasturi Narra
added a comment - ImageSetConfig used for 4.15:
========================
[root@ip-10-0-24-225 tmp]# cat isc415.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.15
minVersion: 4.14.38
maxVersion: 4.15.33
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc415.yaml file://images3 --v2 --authfile auth.json --dry-run 2025/01/09 10:55:18 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 10:55:18 [INFO] : :gear: setting up the environment for you... 2025/01/09 10:55:18 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 10:55:18 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 10:55:18 [INFO] : :mag: collecting release images... 2025/01/09 10:55:35 [INFO] : :mag: collecting operator images... 2025/01/09 10:55:35 [INFO] : :mag: collecting additional images... 2025/01/09 10:55:35 [INFO] : :mag: collecting helm images... 2025/01/09 10:55:36 [WARN] : :warning: 381/382 images necessary for mirroring are not available in the cache. 2025/01/09 10:55:36 [WARN] : List of missing images in : images3/working-dir/dry-run/missing.txt. please re-run the mirror to disk process 2025/01/09 10:55:36 [INFO] : :page_facing_up: list of all images for mirroring in : images3/working-dir/dry-run/mapping.txt 2025/01/09 10:55:36 [INFO] : mirror time : 17.877984399s 2025/01/09 10:55:36 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig used for 4.17:
=========================
[root@ip-10-0-24-225 tmp]# cat isc417.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.17
minVersion: 4.17.0
maxVersion: 4.17.9
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc417.yaml file://images4 --v2 --authfile auth.json --dry-run 2025/01/09 10:59:35 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 10:59:35 [INFO] : :gear: setting up the environment for you... 2025/01/09 10:59:35 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 10:59:35 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 10:59:35 [INFO] : :mag: collecting release images... 2025/01/09 10:59:44 [INFO] : :mag: collecting operator images... 2025/01/09 10:59:44 [INFO] : :mag: collecting additional images... 2025/01/09 10:59:44 [INFO] : :mag: collecting helm images... 2025/01/09 10:59:45 [WARN] : :warning: 381/381 images necessary for mirroring are not available in the cache. 2025/01/09 10:59:45 [WARN] : List of missing images in : images4/working-dir/dry-run/missing.txt. please re-run the mirror to disk process 2025/01/09 10:59:45 [INFO] : :page_facing_up: list of all images for mirroring in : images4/working-dir/dry-run/mapping.txt 2025/01/09 10:59:45 [INFO] : mirror time : 10.146828759s 2025/01/09 10:59:45 [INFO] : :wave: Goodbye, thank you for using oc-mirror
Rama Kasturi Narra
added a comment - ImageSetConfig used for 4.15:
========================
[root@ip-10-0-24-225 tmp]# cat isc415.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.15
minVersion: 4.14.38
maxVersion: 4.15.33
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc415.yaml file: //images3 --v2 --authfile auth.json --dry-run
2025/01/09 10:55:18 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 10:55:18 [INFO] : :gear: setting up the environment for you...
2025/01/09 10:55:18 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 10:55:18 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 10:55:18 [INFO] : :mag: collecting release images...
2025/01/09 10:55:35 [INFO] : :mag: collecting operator images...
2025/01/09 10:55:35 [INFO] : :mag: collecting additional images...
2025/01/09 10:55:35 [INFO] : :mag: collecting helm images...
2025/01/09 10:55:36 [WARN] : :warning: 381/382 images necessary for mirroring are not available in the cache.
2025/01/09 10:55:36 [WARN] : List of missing images in : images3/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
2025/01/09 10:55:36 [INFO] : :page_facing_up: list of all images for mirroring in : images3/working-dir/dry-run/mapping.txt
2025/01/09 10:55:36 [INFO] : mirror time : 17.877984399s
2025/01/09 10:55:36 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig used for 4.17:
=========================
[root@ip-10-0-24-225 tmp]# cat isc417.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.17
minVersion: 4.17.0
maxVersion: 4.17.9
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc417.yaml file: //images4 --v2 --authfile auth.json --dry-run
2025/01/09 10:59:35 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 10:59:35 [INFO] : :gear: setting up the environment for you...
2025/01/09 10:59:35 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 10:59:35 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 10:59:35 [INFO] : :mag: collecting release images...
2025/01/09 10:59:44 [INFO] : :mag: collecting operator images...
2025/01/09 10:59:44 [INFO] : :mag: collecting additional images...
2025/01/09 10:59:44 [INFO] : :mag: collecting helm images...
2025/01/09 10:59:45 [WARN] : :warning: 381/381 images necessary for mirroring are not available in the cache.
2025/01/09 10:59:45 [WARN] : List of missing images in : images4/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
2025/01/09 10:59:45 [INFO] : :page_facing_up: list of all images for mirroring in : images4/working-dir/dry-run/mapping.txt
2025/01/09 10:59:45 [INFO] : mirror time : 10.146828759s
2025/01/09 10:59:45 [INFO] : :wave: Goodbye, thank you for using oc-mirror
Rama Kasturi Narra
added a comment - Verified for 4.13, 4.14, 4.15 and 4.17 releases in the imageSetConfig with the oc-mirror built from the fix PR and i see fix works as expected.
ImageSetConfig used for 4.14:
=========================
[root@ip-10-0-24-225 tmp]# cat isc414.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.14
minVersion: 4.14.38
maxVersion: 4.14.43
shortestPath: true
root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc414.yaml file://images --v2 --authfile auth.json --dry-run 2025/01/09 10:50:22 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 10:50:22 [INFO] : :gear: setting up the environment for you... 2025/01/09 10:50:22 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 10:50:22 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 10:50:22 [INFO] : :mag: collecting release images... 2025/01/09 10:50:22 [INFO] : :mag: collecting operator images... 2025/01/09 10:50:22 [INFO] : :mag: collecting additional images... 2025/01/09 10:50:22 [INFO] : :mag: collecting helm images... 2025/01/09 10:50:22 [WARN] : :warning: 378/380 images necessary for mirroring are not available in the cache. 2025/01/09 10:50:22 [WARN] : List of missing images in : images/working-dir/dry-run/missing.txt. please re-run the mirror to disk process 2025/01/09 10:50:22 [INFO] : :page_facing_up: list of all images for mirroring in : images/working-dir/dry-run/mapping.txt 2025/01/09 10:50:22 [INFO] : mirror time : 785.715093ms 2025/01/09 10:50:22 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig used for 4.13:
=========================
[root@ip-10-0-24-225 tmp]# cat isc413.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.13
minVersion: 4.13.42
maxVersion: 4.13.54
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc413.yaml file://images2 --v2 --authfile auth.json --dry-run 2025/01/09 10:53:31 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 10:53:31 [INFO] : :gear: setting up the environment for you... 2025/01/09 10:53:31 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 10:53:31 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 10:53:31 [INFO] : :mag: collecting release images... 2025/01/09 10:53:57 [INFO] : :mag: collecting operator images... 2025/01/09 10:53:57 [INFO] : :mag: collecting additional images... 2025/01/09 10:53:57 [INFO] : :mag: collecting helm images... 2025/01/09 10:53:58 [WARN] : :warning: 368/368 images necessary for mirroring are not available in the cache. 2025/01/09 10:53:58 [WARN] : List of missing images in : images2/working-dir/dry-run/missing.txt. please re-run the mirror to disk process 2025/01/09 10:53:58 [INFO] : :page_facing_up: list of all images for mirroring in : images2/working-dir/dry-run/mapping.txt 2025/01/09 10:53:58 [INFO] : mirror time : 27.358904112s 2025/01/09 10:53:58 [INFO] : :wave: Goodbye, thank you for using oc-mirror
Rama Kasturi Narra
added a comment - Verified for 4.13, 4.14, 4.15 and 4.17 releases in the imageSetConfig with the oc-mirror built from the fix PR and i see fix works as expected.
ImageSetConfig used for 4.14:
=========================
[root@ip-10-0-24-225 tmp]# cat isc414.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.14
minVersion: 4.14.38
maxVersion: 4.14.43
shortestPath: true
root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc414.yaml file: //images --v2 --authfile auth.json --dry-run
2025/01/09 10:50:22 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 10:50:22 [INFO] : :gear: setting up the environment for you...
2025/01/09 10:50:22 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 10:50:22 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 10:50:22 [INFO] : :mag: collecting release images...
2025/01/09 10:50:22 [INFO] : :mag: collecting operator images...
2025/01/09 10:50:22 [INFO] : :mag: collecting additional images...
2025/01/09 10:50:22 [INFO] : :mag: collecting helm images...
2025/01/09 10:50:22 [WARN] : :warning: 378/380 images necessary for mirroring are not available in the cache.
2025/01/09 10:50:22 [WARN] : List of missing images in : images/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
2025/01/09 10:50:22 [INFO] : :page_facing_up: list of all images for mirroring in : images/working-dir/dry-run/mapping.txt
2025/01/09 10:50:22 [INFO] : mirror time : 785.715093ms
2025/01/09 10:50:22 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig used for 4.13:
=========================
[root@ip-10-0-24-225 tmp]# cat isc413.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.13
minVersion: 4.13.42
maxVersion: 4.13.54
shortestPath: true
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc413.yaml file: //images2 --v2 --authfile auth.json --dry-run
2025/01/09 10:53:31 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 10:53:31 [INFO] : :gear: setting up the environment for you...
2025/01/09 10:53:31 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 10:53:31 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 10:53:31 [INFO] : :mag: collecting release images...
2025/01/09 10:53:57 [INFO] : :mag: collecting operator images...
2025/01/09 10:53:57 [INFO] : :mag: collecting additional images...
2025/01/09 10:53:57 [INFO] : :mag: collecting helm images...
2025/01/09 10:53:58 [WARN] : :warning: 368/368 images necessary for mirroring are not available in the cache.
2025/01/09 10:53:58 [WARN] : List of missing images in : images2/working-dir/dry-run/missing.txt.
please re-run the mirror to disk process
2025/01/09 10:53:58 [INFO] : :page_facing_up: list of all images for mirroring in : images2/working-dir/dry-run/mapping.txt
2025/01/09 10:53:58 [INFO] : mirror time : 27.358904112s
2025/01/09 10:53:58 [INFO] : :wave: Goodbye, thank you for using oc-mirror
Rama Kasturi Narra
added a comment - I am able to successfully reproduce the issue using the steps below:
1. Install RHEL9 cluster with fips enabled
2. Make it STIG complaint by running scripts given by the openshift complaince operator qe team
3. Use latest oc-mirror 4.17
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc.yaml file://images --v2 2025/01/09 09:24:21 [WARN] : :warning: --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready. 2025/01/09 09:24:21 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 09:24:21 [INFO] : :gear: setting up the environment for you... 2025/01/09 09:24:21 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 09:24:21 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 09:24:21 [INFO] : :mag: collecting release images... 2025/01/09 09:24:21 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure 2025/01/09 09:24:21 [ERROR] : generate release signatures: error list invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 2025/01/09 09:24:21 [INFO] : :mag: collecting operator images... 2025/01/09 09:24:21 [INFO] : :mag: collecting additional images... 2025/01/09 09:24:21 [INFO] : :rocket: Start copying the images... 2025/01/09 09:24:21 [INFO] : images to copy 0 2025/01/09 09:24:21 [INFO] : === Results === 2025/01/09 09:24:21 [INFO] : :package: Preparing the tarball archive... 2025/01/09 09:24:21 [INFO] : :wave: Goodbye, thank you for using oc-mirror 2025/01/09 09:24:21 [ERROR] : unable to add cache repositories to the archive : lstat /root/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
4. Build oc-mirror from PR https://github.com/openshift/oc-mirror/pull/997
5. Rerun the mirroring and i see it works.
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc.yaml file://images --v2 --authfile auth.json 2025/01/09 09:53:22 [INFO] : :wave: Hello, welcome to oc-mirror 2025/01/09 09:53:22 [INFO] : :gear: setting up the environment for you... 2025/01/09 09:53:22 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk 2025/01/09 09:53:22 [INFO] : :sleuth_or_spy: going to discover the necessary images... 2025/01/09 09:53:22 [INFO] : :mag: collecting release images... 2025/01/09 09:53:22 [INFO] : :mag: collecting operator images... 2025/01/09 09:53:22 [INFO] : :mag: collecting additional images... 2025/01/09 09:53:22 [INFO] : :mag: collecting helm images... 2025/01/09 09:53:22 [INFO] : :repeat_one: rebuilding catalogs 2025/01/09 09:53:22 [INFO] : :rocket: Start copying the images... 2025/01/09 09:53:22 [INFO] : images to copy 384 ✓ 1/384 : (0s) quay.io/openshift-release-dev/ocp-release:4.16.18-x86_64 :arrow_right: cache ✓ 2/384 : (5s) quay.io/openshift-release-dev/ocp-release:4.16.24-x86_64 :arrow_right: cache ✓ 384/384 : (8s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fefbbd3732f8c3e1674d0db26350821d6d3fe2ea3c146de2272ebacec48b4406 :arrow_right: cache 2025/01/09 10:12:03 [INFO] : === Results === 2025/01/09 10:12:03 [INFO] : :white_check_mark: 384 / 384 release images mirrored successfully 2025/01/09 10:12:03 [INFO] : :package: Preparing the tarball archive... 2025/01/09 10:21:46 [INFO] : mirror time : 28m23.451247132s 2025/01/09 10:21:46 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig that has been used to verify the bug:
=========================================
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.16
minVersion: 4.16.18
maxVersion: 4.16.24
shortestPath: true
Rama Kasturi Narra
added a comment - I am able to successfully reproduce the issue using the steps below:
1. Install RHEL9 cluster with fips enabled
2. Make it STIG complaint by running scripts given by the openshift complaince operator qe team
3. Use latest oc-mirror 4.17
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc.yaml file: //images --v2
2025/01/09 09:24:21 [WARN] : :warning: --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
2025/01/09 09:24:21 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 09:24:21 [INFO] : :gear: setting up the environment for you...
2025/01/09 09:24:21 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 09:24:21 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 09:24:21 [INFO] : :mag: collecting release images...
2025/01/09 09:24:21 [ERROR] : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure
2025/01/09 09:24:21 [ERROR] : generate release signatures: error list invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7
2025/01/09 09:24:21 [INFO] : :mag: collecting operator images...
2025/01/09 09:24:21 [INFO] : :mag: collecting additional images...
2025/01/09 09:24:21 [INFO] : :rocket: Start copying the images...
2025/01/09 09:24:21 [INFO] : images to copy 0
2025/01/09 09:24:21 [INFO] : === Results ===
2025/01/09 09:24:21 [INFO] : : package : Preparing the tarball archive...
2025/01/09 09:24:21 [INFO] : :wave: Goodbye, thank you for using oc-mirror
2025/01/09 09:24:21 [ERROR] : unable to add cache repositories to the archive : lstat /root/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory
4. Build oc-mirror from PR https://github.com/openshift/oc-mirror/pull/997
5. Rerun the mirroring and i see it works.
[root@ip-10-0-24-225 tmp]# ./oc-mirror --config isc.yaml file: //images --v2 --authfile auth.json
2025/01/09 09:53:22 [INFO] : :wave: Hello, welcome to oc-mirror
2025/01/09 09:53:22 [INFO] : :gear: setting up the environment for you...
2025/01/09 09:53:22 [INFO] : :twisted_rightwards_arrows: workflow mode: mirrorToDisk
2025/01/09 09:53:22 [INFO] : :sleuth_or_spy: going to discover the necessary images...
2025/01/09 09:53:22 [INFO] : :mag: collecting release images...
2025/01/09 09:53:22 [INFO] : :mag: collecting operator images...
2025/01/09 09:53:22 [INFO] : :mag: collecting additional images...
2025/01/09 09:53:22 [INFO] : :mag: collecting helm images...
2025/01/09 09:53:22 [INFO] : :repeat_one: rebuilding catalogs
2025/01/09 09:53:22 [INFO] : :rocket: Start copying the images...
2025/01/09 09:53:22 [INFO] : images to copy 384
✓ 1/384 : (0s) quay.io/openshift-release-dev/ocp-release:4.16.18-x86_64 :arrow_right: cache
✓ 2/384 : (5s) quay.io/openshift-release-dev/ocp-release:4.16.24-x86_64 :arrow_right: cache
✓ 384/384 : (8s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fefbbd3732f8c3e1674d0db26350821d6d3fe2ea3c146de2272ebacec48b4406 :arrow_right: cache
2025/01/09 10:12:03 [INFO] : === Results ===
2025/01/09 10:12:03 [INFO] : :white_check_mark: 384 / 384 release images mirrored successfully
2025/01/09 10:12:03 [INFO] : : package : Preparing the tarball archive...
2025/01/09 10:21:46 [INFO] : mirror time : 28m23.451247132s
2025/01/09 10:21:46 [INFO] : :wave: Goodbye, thank you for using oc-mirror
ImageSetConfig that has been used to verify the bug:
=========================================
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v2alpha1
mirror:
platform:
channels:
- name: stable-4.16
minVersion: 4.16.18
maxVersion: 4.16.24
shortestPath: true
More notes for oc-mirror folks. Release images got trusted-key bumps in the OCPBUGS-35528 / cluster-update-keys#57 series to pivot from SHA-1 to SHA-256 self-signatures. I expect oc-mirror needs to do something about your hard-coded trusted-keyrings here and here. It could be "bump those to hard-code the new keyring", which is easy, but leaves you exposed to future similar keyring update issues. Or it could be "restructure to drop the hard-coded keyring, and instead extract the trusted keyring from one of the target release images being mirrored". If you decide to go with the latter, here's what that looks like on the command-line:
$ oc adm release extract --to manifests quay.io/openshift-release-dev/ocp-release:4.16.23-x86_64 # get manifests from the release image Extracted release payload from digest sha256:be725d2e56befbcb28068207b77d731650ad2c82ae77630f46631af750894347 created at 2024-11-14T15:06:33 $ grep -rl release.openshift.io/verification-config-map manifests # find the manifest with this "I'm carrying trusted keys" annotation manifests/0000_90_cluster-update-keys_configmap.yaml $ yaml2json <manifests/0000_90_cluster-update-keys_configmap.yaml | jq -r '.data["verifier-public-key-redhat"]' # get the trusted keyring from the well-known key -----BEGIN PGP PUBLIC KEY BLOCK----- Comment: Use "gpg --dearmor" for unpacking mQINBErgSTsBEACh2A4b0O9t+vzC9VrVtL1AKvUWi9OPCjkvR7Xd8DtJxeeMZ5eF ...
Christopher Novak
added a comment - As a note: This technically will impact more than just U.S Government/DoD customers, as any RHEL 9 box which enforces golang's openpgp to disable SHA-1 signature checking will fail, and require the workaround stated in this ticket.
Christopher Novak
added a comment - As a note: This technically will impact more than just U.S Government/DoD customers, as any RHEL 9 box which enforces golang's openpgp to disable SHA-1 signature checking will fail, and require the workaround stated in this ticket. 
Closing as both related PR's have merged