Value Statement

They suddenly started working, so they can just as easily suddenly stop working. This isn't ideal, and we should investigate what happened so we can guarantee it stays that way.

We were unable to reproduce the issue, however I have looked into several CAPx components and which mirroring issues are occurring in the various pods.

In all of the below cases, an ImageDigestMirrorSet is put in place to attempt to reroute from registry.redhat.io to quay.io/acm-d when necessary, so all "back-off pulling image" errors include attempts to pull from quay.io as well

mce-capi-webhook-config

Back-off pulling image "registry.redhat.io/multicluster-engine/cluster-api-webhook-config-rhel9@sha256:993230c95b9dc5003659aecc152d52dbb053f381d3eb43ac72f4bb35b92a7230"
This whole image does not appear to exist at registry.redhat.io, nor is it mirrored into quay.io, it's not just a missing SHA/tag. The CPaaS build does not appear to create this pod at all (as of 2.14.0-DOWNSTREAM-2025-06-05-04-29-38)

capoa-bootstrap-controller-manager

Back-off pulling image "registry.redhat.io/multicluster-engine/capoa-bootstrap-rhel9@sha256:19542174a5484114ef4a8150ed61d7ef85b5847a4cbbf3ac0f921d66a613f3dc"
This image exists in quay.io but not registry.redhat.io, however even in quay.io this sha does not exist, so it cannot pull. Under CPaaS, it pulls registry.redhat.io/multicluster-engine/capoa-bootstrap-rhel9@sha256:e24e50d0183ca1e206b7edf349cac61bdc4ab68e20e4673bace7996508d794b5 instead, which does exist, so it works.

capoa-controlplane-controller-manager

Back-off pulling image "registry.redhat.io/multicluster-engine/capoa-control-plane-rhel9@sha256:b3f032b409969a117587b7547f2d47291e9f8b78bc00457cf6c2d9e516e640fa"
Again, this image exists in quay.io, but not registry.redhat.io and that particular sha is not included in the list of images on quay.io, so it fails. The latest sha is sha256:74bd51ac29222d1f9419d3637764b74887fdff382c47ab5d310d47be9c3310ac, which is what the CPaaS build uses and why it works.

capi-controller-manager

Succeeded in pulling quay.io/acm-d/ose-cluster-api-rhel9@sha256:0dbcbb6e334cd5c9cfb4adbf8548f4bb27a7aa40d0d2f4bee7275c8b72508486 when asked to mirror in both Konflux and CPaaS

Definition of Done for Engineering Story Owner (Checklist)

• ...

Development Complete

• The code is complete.
• Functionality is working.
• Any required downstream Docker file changes are made.

Tests Automated

• [ ] Unit/function tests have been automated and incorporated into the
  build.
• [ ] 100% automated unit/function test coverage for new or changed APIs.

Secure Design

• [ ] Security has been assessed and incorporated into your threat model.

Multidisciplinary Teams Readiness

• [ ] Create an informative documentation issue using the [Customer
  Portal_doc_issue template](
  https://github.com/stolostron/backlog/issues/new?assignees=&labels=squad%3Adoc&template=doc_issue.md&title=),
  and ensure doc acceptance criteria is met. Link the development issue to
  the doc issue.
• [ ] Provide input to the QE team, and ensure QE acceptance criteria
  (established between story owner and QE focal) are met.

Support Readiness

• [ ] The must-gather script has been updated.