Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-66414

MicroShift Cert-Manager Multi-Arch Digest Issues

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • 4.21.0
    • 4.20
    • MicroShift
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      TL,DR: cert-manager images in MicroShift 4.20 use multi-arch manifest lists, causing runtime digest mismatches or significant storage bloat in offline bootc builds.

      Description of problem:

      We are building bootc-based RHEL 9.6 / MicroShift 4.20 images for air-gapped (offline) environments. The cert-manager Operator manifests shipped with MicroShift 4.20 reference images using their Manifest List (multi-arch) digests.
      
      This creates a critical issue for offline builds, identical to the behavior seen with lvms-operator in OCPBUGS-54779 (and similar to OCPBUGS-51329).   

      Version-Release number of selected component (if applicable):

      MicroShift 4.20
      
      [root@ushift05 ~]# dnf info microshift-cert-manager
      Updating Subscription Management repositories.This system has release set to 9.6 and it receives updates only for this release.Last metadata expiration check: 1:48:05 ago on Thu 04 Dec 2025 03:09:09 PM CET.
      Installed Packages
      Name         : microshift-cert-manager
      Version      : 4.20.0
      Release      : 202510201126.p0.g1c4675a.assembly.4.20.0.el9
      Architecture : x86_64
      Size         : 1.1 M
      Source       : microshift-4.20.0-202510201126.p0.g1c4675a.assembly.4.20.0.el9.src.rpm
      Repository   : @System
      From repo    : rhocp-4.20-for-rhel-9-x86_64-rpms
      Summary      : Cert Manager for MicroShift
      URL          : https://github.com/openshift/microshift
      License      : ASL 2.0
      Description  : [Maintainer] project: OCPBUGS, component: MicroShift
                   : The microshift-cert-manager package provides the required manifests for the Cert Manager to be installed on MicroShif
      
      [root@ushift05 ~]# dnf info microshift-cert-manager-release-info
      Updating Subscription Management repositories.This system has release set to 9.6 and it receives updates only for this release.Last metadata expiration check: 1:48:36 ago on Thu 04 Dec 2025 03:09:09 PM CET.
      Installed Packages
      Name         : microshift-cert-manager-release-info
      Version      : 4.20.0
      Release      : 202510201126.p0.g1c4675a.assembly.4.20.0.el9
      Architecture : noarch
      Size         : 2.1 k
      Source       : microshift-4.20.0-202510201126.p0.g1c4675a.assembly.4.20.0.el9.src.rpm
      Repository   : @System
      From repo    : rhocp-4.20-for-rhel-9-x86_64-rpms
      Summary      : Release information for Cert Manager for MicroShift
      URL          : https://github.com/openshift/microshift
      License      : ASL 2.0
      Description  : [Maintainer] project: OCPBUGS, component: MicroShift
                   : The microshift-cert-manager-release-info package provides release information files for this
                   : release. These files contain the list of container image references used by Cert Manager
                   : and can be used to embed those images into osbuilder blueprints or bootc containerfiles.
          

      How reproducible:

      Scenario A (Runtime Failure): If we embed images using skopeo copy docker://... dir://..., Skopeo resolves the Manifest List to the specific architecture (e.g., linux/amd64) and discards the Manifest List index.
        - Result: The offline storage contains the image with the AMD64 digest.
        - Failure: At runtime, the cert-manager deployment requests the Manifest List digest. CRI-O cannot find this digest locally, attempts to pull it from the internet, and fails with ImagePullBackOff / i/o timeout.
      
      Scenario B (Storage Bloat): If we use skopeo copy --all --preserve-digests (the current workaround), Skopeo preserves the Manifest List digest.
        - Result: Runtime works because the digest matches.
        - Failure: Skopeo downloads layers for ALL architectures (arm64, ppc64le, s390x), causing significant and unnecessary storage bloat on the target edge device.     

      Steps to Reproduce:

      1. On a RHEL 9 / MicroShift 4.20 amd64 host.
      2. Attempt to embed cert-manager images into a bootc container image.
      3. Method 1: Run skopeo copy docker://<image>@<manifest-list-digest> dir://... without --all.
         - Result: Build succeeds, but MicroShift pod fails to start offline due to digest mismatch.
      4. Method 2: Run skopeo copy --all docker://<image>@<manifest-list-digest> dir://...
         - Result: Runtime works, but storage usage spikes due to unused non-amd64 layers.

      Actual results:

      The cert-manager operator strictly enforces the multi-arch digest, forcing users to either suffer runtime failures or consume excessive disk space by mirroring all architectures.    

      Expected results:

      MicroShift should allow the use of single-architecture local images for cert-manager without requiring the presence of the full upstream Manifest List, or provide a mechanism to pin the deployment to the local architecture's digest during installation.    

      Additional info:

      Operator Image (Manifest List): registry.redhat.io/cert-manager/cert-manager-operator-rhel9@sha256:4d5e238300ce6f427a1045d51d6b37a4e5c5633985208ebb44f91e7dd53897d9
      
      Resolved AMD64 Image: sha256:0431bde46dd25e63db6e3d2d5395d6d27f44b6e0b0bee1a4dd65137152d6c650    

              microshift-team MicroShift Team
              rhn-support-arolivei Arthur Oliveira
              None
              None
              John George John George
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: