OpenShift 4.14.14 on s390x

RedHat Quay v3.10.3

After successfully mirroring the IBM MQ operator via the oc-mirror command to Red Hat Quay and applying the newly generated IBM CatalogSource YAML file. The created catalog pod in the openShift-marketplace namespace enters CrashLoopBackOff.

The pod's log contains the following error:

time="2024-04-22T21:04:29Z" level=info msg="starting pprof endpoint" address="localhost:6060"time="2024-04-22T21:04:30Z"
 level=fatal msg="cache requires rebuild: cache reports digest as 
\"53dccfc008139021\", but computed digest is \"1b1bc17636fe84b1\""   


Problem Overview:

Download and use the oc-mirror plugin provided by 4.14.14-rc.4 for s390x located here: (https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.14.14/oc-mirror.tar.gz)

Download and use the oc cli for OpenShift 4.14 on s390x here:
(https://mirror.openshift.com/pub/openshift-v4/s390x/clients/ocp/4.14.14/openshift-client-linux-4.14.14.tar.gz)

$ oc version 

Client Version: 4.14.14 Kustomize Version: v5.0.1 Server Version: 4.14.14 Kubernetes Version: v1.27.10+28ed2d7 

$ oc-mirror version 

Logging to .oc-mirror.log WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.14.0-202402081809.p0.g6874730.assembly.stream.el8-6874730", GitCommit:"68747309c90d1c64d5404ec9ea12bc7c36a80555", GitTreeState:"clean", BuildDate:"2024-02-08T23:49:25Z", GoVersion:"go1.20.12 X:strictfipsruntime", Compiler:"gc", Platform:"linux/s390x"}

$ cat imageset-config.yaml

kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  registry:
    imageURL: quayreg1.fpet.pokprv.stglabs.ibm.com/test0/oc-mirror-meta
    skipTLS: true
mirror:
  platform:
    architectures:
      - "s390x"
  operators:
    - catalog: icr.io/cpopen/ibm-operator-catalog:v1.27
      packages:
      - name: ibm-mq
        channels:
        - name: v3.1

$ oc mirror --config=./imageset-config.yaml  docker://quayreg1.fpet.pokprv.stglabs.ibm.com/test0/operators

...

info: Mirroring completed in 2.97s (63.85MB/s)
Rendering catalog image "quayreg1.fpet.pokprv.stglabs.ibm.com/test0/operators/cpopen/ibm-operator-catalog:v1.27" with file-based catalog
Writing image mapping to oc-mirror-workspace/results-1713822225/mapping.txt
Writing CatalogSource manifests to oc-mirror-workspace/results-1713822225
Writing ICSP manifests to oc-mirror-workspace/results-1713822225

$ ls 
catalogSource-cs-ibm-operator-catalog.yaml

$ oc apply -f catalogSource-cs-ibm-operator-catalog.yaml
catalogsource.operators.coreos.com/cs-ibm-operator-catalog created

$ oc get pods -n openshift-marketplace
NAME                                                              READY   STATUS             RESTARTS        AGE
...

cs-ibm-operator-catalog-tm2m6                                     0/1     CrashLoopBackOff   10 (114s ago)   28m
           0               33d


$ oc describe pod cs-ibm-operator-catalog-tm2m6 -n openshift-marketplace | grep -a10 -i error
Containers:
  registry-server:
    Container ID:  cri-o://6a72df4c9173a2e5b4d5d3c555c76bc35d5ea27c19648077923ea9be3552b21f
    Image:         quayreg1.fpet.pokprv.stglabs.ibm.com/test0/operators/cpopen/ibm-operator-catalog:v1.27
    Image ID:      quayreg1.fpet.pokprv.stglabs.ibm.com/test0/operators/cpopen/ibm-operator-catalog@sha256:5868aad164a130d4e646762f0463d0e4d01205a90844018b24b09b2f89d68cfe
    Port:          50051/TCP
    Host Port:     0/TCP
    State:         Waiting
      Reason:      CrashLoopBackOff
    Last State:    Terminated
      Reason:      Error
      Message:     time="2024-04-22T22:16:56Z" level=info msg="starting pprof endpoint" address="localhost:6060"
time="2024-04-22T22:16:57Z" level=fatal msg="cache requires rebuild: cache reports digest as \"53dccfc008139021\", but computed digest is \"1b1bc17636fe84b1\""      Exit Code:    1
      Started:      Mon, 22 Apr 2024 18:16:55 -0400
      Finished:     Mon, 22 Apr 2024 18:16:57 -0400
    Ready:          False
    Restart Count:  10
    Requests:
      cpu:        10m

 4.14.14

 Always

    1. Run OpenShift 4.14.14 on s390x
    2. Mirror the IBM MQ operator from icr.io/cpopen/ibm-operator-catalog using  the oc-mirror plugin to Red Hat quay registry
    3. Apply the newly created catalog source
    

Newly created Catalog pod in the openShift-marketplace enters CrashLoopBackOff with strange "cache requires rebuild: " error.

Newly created Catalog pod in the openShift-marketplace should be running without failure 

This problem appears to be cited in an existing RedHat solution here: (https://access.redhat.com/solutions/7041232)

However, the solution provided suggests running the latest oc-mirror command. Looking at my environment above, given the version of OpenShift I'm running (4.14.4), I believe to be already using the latest oc-mirror command.