Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-48695

OLMv0: excessive catalog source snapshots cause severe performance regression [openshift-4.17.z]

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 4.15.z, 4.17.z, 4.16.z, 4.18.0
    • OLM
    • Critical
    • Yes
    • Eevee OLM Sprint 265
    • 1
    • Proposed
    • False
    • Hide

      None

      Show
      None
    • Addresses issue where excessive catalog source snapshots were causing severe performance regressions.
    • Bug Fix
    • In Progress

      This is a clone of issue OCPBUGS-48644. The following is the description of the original issue:

      Description of problem:

      OLM's catalog operator makes excessive and unnecessary GRPC API requests to catalog source pods every time a subscription is reconciled. The observable outcome is that catalog pods have very high CPU usage, and depending on the number of subscriptions and the state of the system, the high CPU usage may be sustained permanently.    

      Version-Release number of selected component (if applicable):

      4.15

      How reproducible:

      100%    

      Steps to Reproduce:

          1. Get credentials for a 4.15 or higher cluster
          2. Install an operator via a subscription against any catalog source
          3. (On 4.17 or higher only) Check the logs of the catalog pod that serves the catalog, and see that it is receiving many ListBundles and GetPackage requests in a short period of time, especially when a subscription is performing an installation or upgrade.
          

      Actual results:

      ~20 ListBundles calls performed while installing a single operator via a subscription

      Expected results:

       1 ListBundles call performed while installing a single operator via a subscription (other subscription reconciles should use a cached result until the cache is invalidated)

      Additional info:

          

              rh-ee-cchantse Catherine Chan-Tse
              openshift-crt-jira-prow OpenShift Prow Bot
              Kui Wang Kui Wang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: