Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-37667

IBM Operator Index Image fails with "cache requires rebuild: cache reports digest as xxx, but computed digest is yyy"

XMLWordPrintable

    • +
    • Critical
    • None
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, when users mirrored operator catalogs with the `--rebuild-catalogs` flag, the catalog cache was regenerated locally, which caused failures either due to compatibility issues with the `opm` binary and the platform or due to cache integrity problems on the cluster. With this update, the `--rebuild-catalogs` flag defaults to true, so catalogs are rebuilt without regenerating the internal cache. Additionally, the image command has been modified to generate the cache during pod startup, which may delay pod initialization.
      (link:https://issues.redhat.com/browse/OCPBUGS-37667[*OCPBUGS-37667*])
      Show
      Previously, when users mirrored operator catalogs with the `--rebuild-catalogs` flag, the catalog cache was regenerated locally, which caused failures either due to compatibility issues with the `opm` binary and the platform or due to cache integrity problems on the cluster. With this update, the `--rebuild-catalogs` flag defaults to true, so catalogs are rebuilt without regenerating the internal cache. Additionally, the image command has been modified to generate the cache during pod startup, which may delay pod initialization. (link: https://issues.redhat.com/browse/OCPBUGS-37667 [* OCPBUGS-37667 *])
    • Bug Fix
    • Done

      Description of problem:

      After successfully mirroring the ibm-ftm-operator via the latest oc-mirror command to internal registry and applying the newly generated IBM CatalogSource YAML file. The created catalog pod in the openShift-marketplace namespace enters CrashLoopBackOff.
      
      Customer is trying to mirror operators and  list the catalogue the command has no issues, but catalog pod is crashing with the following error:
      ~~~
      time="2024-07-10T13:43:07Z" level=info msg="starting pprof endpoint" address="localhost:6060"
      time="2024-07-10T13:43:08Z" level=fatal msg="cache requires rebuild: cache reports digest as \"e891bfd5a4cb5702\", but computed digest is \"1922475dc0ee190c\""
      ~~~
      
          

      Version-Release number of selected component (if applicable):

      oc-mirror 4.16
      OCP 4.14.z
      
          

      How reproducible:

      
      
          

      Steps to Reproduce:

          1. Create catalog image with the following imagesetconfiguration:
      ~~~
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      archiveSize: 4
      storageConfig:
        registry:
          imageURL: <internal-registry>:Port/oc-mirror-metadata/12july24
          skipTLS: false
      mirror:
        platform:
          architectures:
            - "amd64"
          channels:
          - name: stable-4.14
            minVersion: 4.14.11
            maxVersion: 4.14.30
            type: ocp
            shortestPath: true
          graph: true
        operators:
        - catalog: icr.io/cpopen/ibm-operator-catalog:v1.22
          packages:
          - name: ibm-ftm-operator
            channels:
            - name: v4.4
      ~~~
          2.  Run the following command:
      ~~~
      /oc-mirror --config=./imageset-config.yaml docker://Internal-registry:Port --rebuild-catalogs
      ~~~
          3. Create catalogsourcepod under openshift-marketplace namespace:
      ~~~
       cat oc-mirror-workspace/results-1721222945/catalogSource-cs-ibm-operator-catalog.yaml
      apiVersion: operators.coreos.com/v1alpha1
      kind: CatalogSource
      metadata:
        name: cs-ibm-operator-catalog
        namespace: openshift-marketplace
      spec:
        image: Internal-registry:Port/cpopen/ibm-operator-catalog:v1.22
        sourceType: grpc
      ~~~
          

      Actual results:

      
      catalog pod is crashing with the following error:
      ~~~
      time="2024-07-10T13:43:07Z" level=info msg="starting pprof endpoint" address="localhost:6060"
      time="2024-07-10T13:43:08Z" level=fatal msg="cache requires rebuild: cache reports digest as \"e891bfd5a4cb5702\", but computed digest is \"1922475dc0ee190c\""
      ~~~
      
          

      Expected results:

      The pod should run without any issue. 
      
          

      Additional info:

      1. The issue is reproducible with the OCP 4.14.14 and OCP 4.14.29
      2. Customer is already using oc-mirror 4.16:
      ~~~
      ./oc-mirror version
      WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
      Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.16.0-202407030803.p0.g394b1f8.assembly.stream.el9-394b1f8", GitCommit:"394b1f814f794f4f01f473212c9a7695726020bf", GitTreeState:"clean", BuildDate:"2024-07-03T10:18:49Z", GoVersion:"go1.21.11 (Red Hat 1.21.11-1.module+el8.10.0+21986+2112108a) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
      ~~~
      3. Customer tried with workaround described in the KB[1]: https://access.redhat.com/solutions/7006771 but no luck
      4. Customer also tried to set the OPM_BINARY, but didn't work. They download  OPM with respective arch: https://github.com/operator-framework/operator-registry/releases rename the downloaded binary as opm and set below variable before executing oc-mirror
      OPM_BINARY=/path/to/opm
      
          

              luzuccar@redhat.com Luigi Mario Zuccarelli
              rhn-support-mbagga Mithilesh Bagga
              ying zhou ying zhou
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: