Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-37667

IBM Operator Index Image fails with "cache requires rebuild: cache reports digest as xxx, but computed digest is yyy"

XMLWordPrintable

    • Critical
    • None
    • False
    • Hide

      None

      Show
      None
    • Hide
      when the user mirrors operator catalogs with flag `--rebuild-catalogs` set, catalog cache was regenerated on the local machine. This required extraction and use of an `opm` binary from the catalog image, and lead to either failures in mirroring (due to mismatch between `opm` binary's supported OS and platform and the platform+ OS runnning oc-mirror), or to failures in running the catalog source on the cluster (due to cache integrity issues).
      with this fix, `--rebuild-catalogs` default value is true: so if omitted, catalog will be rebuilt, but without regenerating its internal cache. Also, the CMD of the image is modified from `opm serve /configs --cache-dir=/tmp/cache` to `opm serve /configs`. This will force the pod to generate the cache at startup, and may lead to delays in pod startup.
      Show
      when the user mirrors operator catalogs with flag `--rebuild-catalogs` set, catalog cache was regenerated on the local machine. This required extraction and use of an `opm` binary from the catalog image, and lead to either failures in mirroring (due to mismatch between `opm` binary's supported OS and platform and the platform+ OS runnning oc-mirror), or to failures in running the catalog source on the cluster (due to cache integrity issues). with this fix, `--rebuild-catalogs` default value is true: so if omitted, catalog will be rebuilt, but without regenerating its internal cache. Also, the CMD of the image is modified from `opm serve /configs --cache-dir=/tmp/cache` to `opm serve /configs`. This will force the pod to generate the cache at startup, and may lead to delays in pod startup.
    • Bug Fix
    • In Progress

      Description of problem:

      After successfully mirroring the ibm-ftm-operator via the latest oc-mirror command to internal registry and applying the newly generated IBM CatalogSource YAML file. The created catalog pod in the openShift-marketplace namespace enters CrashLoopBackOff.
      
      Customer is trying to mirror operators and  list the catalogue the command has no issues, but catalog pod is crashing with the following error:
      ~~~
      time="2024-07-10T13:43:07Z" level=info msg="starting pprof endpoint" address="localhost:6060"
      time="2024-07-10T13:43:08Z" level=fatal msg="cache requires rebuild: cache reports digest as \"e891bfd5a4cb5702\", but computed digest is \"1922475dc0ee190c\""
      ~~~
      
          

      Version-Release number of selected component (if applicable):

      oc-mirror 4.16
      OCP 4.14.z
      
          

      How reproducible:

      
      
          

      Steps to Reproduce:

          1. Create catalog image with the following imagesetconfiguration:
      ~~~
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      archiveSize: 4
      storageConfig:
        registry:
          imageURL: <internal-registry>:Port/oc-mirror-metadata/12july24
          skipTLS: false
      mirror:
        platform:
          architectures:
            - "amd64"
          channels:
          - name: stable-4.14
            minVersion: 4.14.11
            maxVersion: 4.14.30
            type: ocp
            shortestPath: true
          graph: true
        operators:
        - catalog: icr.io/cpopen/ibm-operator-catalog:v1.22
          packages:
          - name: ibm-ftm-operator
            channels:
            - name: v4.4
      ~~~
          2.  Run the following command:
      ~~~
      /oc-mirror --config=./imageset-config.yaml docker://Internal-registry:Port --rebuild-catalogs
      ~~~
          3. Create catalogsourcepod under openshift-marketplace namespace:
      ~~~
       cat oc-mirror-workspace/results-1721222945/catalogSource-cs-ibm-operator-catalog.yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: cs-ibm-operator-catalog
        namespace: openshift-marketplace
      spec:
        image: Internal-registry:Port/cpopen/ibm-operator-catalog:v1.22
        sourceType: grpc
      ~~~
          

      Actual results:

      
      catalog pod is crashing with the following error:
      ~~~
      time="2024-07-10T13:43:07Z" level=info msg="starting pprof endpoint" address="localhost:6060"
      time="2024-07-10T13:43:08Z" level=fatal msg="cache requires rebuild: cache reports digest as \"e891bfd5a4cb5702\", but computed digest is \"1922475dc0ee190c\""
      ~~~
      
          

      Expected results:

      The pod should run without any issue. 
      
          

      Additional info:

      1. The issue is reproducible with the OCP 4.14.14 and OCP 4.14.29
      2. Customer is already using oc-mirror 4.16:
      ~~~
      ./oc-mirror version
      WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
      Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.16.0-202407030803.p0.g394b1f8.assembly.stream.el9-394b1f8", GitCommit:"394b1f814f794f4f01f473212c9a7695726020bf", GitTreeState:"clean", BuildDate:"2024-07-03T10:18:49Z", GoVersion:"go1.21.11 (Red Hat 1.21.11-1.module+el8.10.0+21986+2112108a) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
      ~~~
      3. Customer tried with workaround described in the KB[1]: https://access.redhat.com/solutions/7006771 but no luck
      4. Customer also tried to set the OPM_BINARY, but didn't work. They download  OPM with respective arch: https://github.com/operator-framework/operator-registry/releases rename the downloaded binary as opm and set below variable before executing oc-mirror
      OPM_BINARY=/path/to/opm
      
          

            luzuccar@redhat.com Luigi Mario Zuccarelli
            rhn-support-mbagga Mithilesh Bagga
            ying zhou ying zhou
            Votes:
            1 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: