Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25622

"community-operators" Pods are crashlooping due to "cache requires rebuild" error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • 4.12.z, 4.12
    • OLM
    • Important
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Customer has reported that on all (connected) OpenShift Container Platform clusters since roughly 15:45 CET (14:45 UTC) they are seeing the "community-operators" Pods repeatedly crashing:
      
      ---
       oc get pods -n openshift-marketplace
      NAME                                                              READY   STATUS             RESTARTS        AGE
      2d34b4ff36cb605756f58f2605d02eb7d37a79e9bc1f45ef4d7aa21387x22vs   0/1     Completed          0               4d21h
      b77d34e52e5eef9d4957670dd591057eafd230669cdf32a143d28c2e9b6m84d   0/1     Completed          0               4d21h
      certified-operators-hmdwn                                         1/1     Running            0               135m
      community-operators-bsrls                                         1/1     Running            0               6h21m
      community-operators-rfr76                                         0/1     CrashLoopBackOff   9 (4m33s ago)   25m
      marketplace-operator-c67ccf5d6-9p8mw                              1/1     Running            0               32d
      redhat-marketplace-252cg                                          1/1     Running            0               2d23h
      redhat-operators-r5tdl
      ---
      
      The logs show the following:
      
      ---
      time="2023-12-18T15:11:26Z" level=fatal msg="cache requires rebuild: cache reports digest as \"171343b3bec8c85c\", but computed digest is \"4551d160b0bf9d6d\""
      ---
      
      The same was also reported internally: https://redhat-internal.slack.com/archives/C3VS0LV41/p1702911540727929

      Version-Release number of selected component (if applicable):

      OpenShift Container Platform 4.12.41

      How reproducible:

      Since this afternoon (CET)

      Steps to Reproduce:

      0. Ensure OpenShift Container Platform cluster is connected (not disconnected)
      1. Observe Pods in "openshift-marketplace" namespace

      Actual results:

      "community-operators" Pods are crashing with "cache requires rebuild: cache reports digest [..]" error

      Expected results:

      No Pods are in CrashLoopBackOff

      Additional info:

      Possible workaround is discussed in https://redhat-internal.slack.com/archives/C3VS0LV41/p1702913235898179?thread_ts=1702911540.727929&cid=C3VS0LV41

              agreene1991 Alexander Greene (Inactive)
              rhn-support-skrenger Simon Krenger
              Xia Zhao Xia Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: