Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-48363

oc-mirror V2 fails on FIPS enabled and STIG compliant RHEL 9 system

XMLWordPrintable

    • Critical
    • None
    • CLID Sprint 265
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      Release images are now signed by a new sha256 trusted-key that is different from SHA-1 key used previously.
      On RHEL9 FIPS STIG compliant machines, verification of release signature by the old SHA-1 key was failing, because of restriction to use weak keys.
      The fix consists in changing the key used by oc-mirror for release signature verification.
      Show
      Release images are now signed by a new sha256 trusted-key that is different from SHA-1 key used previously. On RHEL9 FIPS STIG compliant machines, verification of release signature by the old SHA-1 key was failing, because of restriction to use weak keys. The fix consists in changing the key used by oc-mirror for release signature verification.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-48314. The following is the description of the original issue:

      This is a clone of issue OCPBUGS-47453. The following is the description of the original issue:

      Description of problem:

          When running oc-mirror V2 (either 4.16 or 4.17 has been tested) on a RHEL 9 FIPS enabled and STIG Security profile enforced system, oc-mirror fails due to a hard coded PGP key in oc-mirror V2.

      Version-Release number of selected component (if applicable):

          At least 4.16-4.17

      How reproducible:

          Very reproducible

      Steps to Reproduce:

      1. Install latest oc-mirror
      2. Create a cluster-images.yml file
      ImageSetConfiguration
      apiVersion: mirror.openshift.io/v2alpha1
      mirror:
        platform:
          channels:
          - name: stable-4.16
            minVersion: 4.16.18
            maxVersion: 4.16.24
            shortestPath: true
          

      3. run oc-mirror with the following flags:

      [cnovak@localhost ocp4-disconnected-config]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v2
      
      2024/12/18 14:40:01  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
      2024/12/18 14:40:01  [INFO]   : 👋 Hello, welcome to oc-mirror
      2024/12/18 14:40:01  [INFO]   : ⚙️  setting up the environment for you...
      2024/12/18 14:40:01  [INFO]   : 🔀 workflow mode: mirrorToDisk 
      2024/12/18 14:40:01  [INFO]   : 🕵️  going to discover the necessary images...
      2024/12/18 14:40:01  [INFO]   : 🔍 collecting release images...
      2024/12/18 14:40:02  [ERROR]  : openpgp: invalid data: user ID self-signature invalid: openpgp: invalid signature: RSA verification failure
      2024/12/18 14:40:02  [ERROR]  : generate release signatures: error list invalid signature for 3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 image quay.io/openshift-release-dev/ocp-release@sha256:3f14e29f5b42e1fee7d7e49482cfff4df0e63363bb4a5e782b65c66aba4944e7 
      2024/12/18 14:40:02  [INFO]   : 🔍 collecting operator images...
      2024/12/18 14:40:02  [INFO]   : 🔍 collecting additional images...
      2024/12/18 14:40:02  [INFO]   : 🚀 Start copying the images...
      2024/12/18 14:40:02  [INFO]   : images to copy 0 
      2024/12/18 14:40:02  [INFO]   : === Results ===
      2024/12/18 14:40:02  [INFO]   : 📦 Preparing the tarball archive...
      2024/12/18 14:40:02  [INFO]   : 👋 Goodbye, thank you for using oc-mirror
      2024/12/18 14:40:02  [ERROR]  : unable to add cache repositories to the archive : lstat /home/cnovak/.oc-mirror/.cache/docker/registry/v2/repositories: no such file or directory 

      Expected results/immediate workaround:

      [cnovak@localhost ~]$ curl -s https://raw.githubusercontent.com/openshift/cluster-update-keys/d44fca585d081a72cb2c67734556a27bbfc9470e/manifests.rhel/0000_90_cluster-update-keys_configmap.yaml | sed -n '/openshift[.]io/d;s/Comment:.*//;s/^    //p' > /tmp/pgpkey
      [cnovak@localhost ~]$ export OCP_SIGNATURE_VERIFICATION_PK=/tmp/pgpkey
      [cnovak@localhost ~]$ /pods/content/bin/oc-mirror --config /pods/content/images/cluster-images.yml file:///pods/content/images/cluster-images --v22024/12/19 08:54:42  [WARN]   : ⚠️  --v2 flag identified, flow redirected to the oc-mirror v2 version. This is Tech Preview, it is still under development and it is not production ready.
      2024/12/19 08:54:42  [INFO]   : 👋 Hello, welcome to oc-mirror
      2024/12/19 08:54:42  [INFO]   : ⚙️  setting up the environment for you...
      2024/12/19 08:54:42  [INFO]   : 🔀 workflow mode: mirrorToDisk 
      2024/12/19 08:54:42  [INFO]   : 🕵️  going to discover the necessary images...
      2024/12/19 08:54:42  [INFO]   : 🔍 collecting release images...
      2024/12/19 08:54:42  [INFO]   : 🔍 collecting operator images...
      2024/12/19 08:54:42  [INFO]   : 🔍 collecting additional images...
      2024/12/19 08:54:42  [INFO]   : 🚀 Start copying the images...
      2024/12/19 08:54:42  [INFO]   : images to copy 382 
       ⠸   1/382 : (7s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb 
       ✓   2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace 
       ✓   3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 
       ✓   4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 
       ⠋   1/382 : (19s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:32f80a2ee0f52e0c07a6790171be70a1b92010d8d395e9e14b4ee5f268e384bb 
       ✓   2/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a61b758c659f93e64d4c13a7bbc6151fe8191c2421036d23aa937c44cd478ace 
       ✓   3/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29ba4e3ff278741addfa3c670ea9cc0de61f7e6265ebc1872391f5b3d58427d0 
       ✓   4/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2809165826b9094873f2bc299a28980f92d7654adb857b73463255eac9265fd8 
       ✓   5/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e54fc21197c341fe257d2f2f2ad14b578483c4450474dc2cf876a885f11e745 
       ✓   6/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c934b4d95545e29f9cb7586964fd43cdb7b8533619961aaa932fe2923ab40db 
       ✓   7/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:851ba9ac5219a9f11e927200715e666ae515590cd9cc6dde9631070afb66b5d7 
       ✓   8/382 : (1s) quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f614ef855220f2381217c31b8cb94c05ef20edf3ca23b5efa0be1b957cdde3a4 

      Additional info:

      The reason this is a critical issue, is Red Hat has a relatively large footprint within the DoD/U.S Government space, and anyone who is working in a disconnected environment, with a STIG Policy enforced on a RHEL 9 machine, will run into this problem.
      
      
      Additionally, below is output from oc-mirror version
      
      
      
      [cnovak@localhost ~]$ oc-mirror version
      WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
      Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.17.0-202411251634.p0.g07714b7.assembly.stream.el9-07714b7", GitCommit:"07714b7c836ec3ad1b776f25b44c3b2c2f083aa2", GitTreeState:"clean", BuildDate:"2024-11-26T08:28:42Z", GoVersion:"go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

              luzuccar@redhat.com Luigi Mario Zuccarelli
              openshift-crt-jira-prow OpenShift Prow Bot
              ying zhou ying zhou
              Christopher Novak, Dan Clark, Mark Salowitz, Matthew Riensch, W. Trevor King
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: