Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-18469

Azure Image Registry Operator Making too Many Storage Account List Calls

    XMLWordPrintable

Details

    • No
    • Sprint 241, Sprint 242
    • 2
    • False
    • Hide

      None

      Show
      None
    • Hide
      * The Image Registry Operator makes API calls to the storage account list endpoint as part of obtaining access keys. In projects with several {product-title} clusters, this might lead to API limits being reached. As a result, `429` errors were returned when attempting to create new clusters. With this update, the time between calls has been increased from 5 minutes to 20 minutes, and API limits are no longer reached. (link:https://issues.redhat.com/browse/OCPBUGS-18469[*OCPBUGS-18469*])
      Show
      * The Image Registry Operator makes API calls to the storage account list endpoint as part of obtaining access keys. In projects with several {product-title} clusters, this might lead to API limits being reached. As a result, `429` errors were returned when attempting to create new clusters. With this update, the time between calls has been increased from 5 minutes to 20 minutes, and API limits are no longer reached. (link: https://issues.redhat.com/browse/OCPBUGS-18469 [* OCPBUGS-18469 *])
    • Bug Fix
    • Done

    Description

      Description of problem:

      The image registry operator in Azure by default has two replicas.  Every 5 minutes, each of those replicas makes a call to the StorageAccount List operation for the image registry storage account.  
      
      Azure has published limits for storage account throttling operations.  These limits are 100 calls to list operations every 5 minutes based on the subscription & region pair that exists. 
      
      Because of this, customers are limited to <50 clusters per subscription and region in Azure.  This number can change based on the number of image registry replicas as well as customer activity on List storage account operations within that subscription and region.  
      
      On Azure Red Hat OpenShift managed service, we occasionally have customers exceeding these limits including internal customers for demos, preventing them from creating new clusters within the subscription & region due to these scaling limits.  
      

      Version-Release number of selected component (if applicable):

      N/A
      

      How reproducible:

      Always.  
      

      Steps to Reproduce:

      1. Scale up the number of image registry pods to hit the 100 / 5 minute List limit (50 replicas, or enough clusters within a given subscription & region)
      2. Attempt to create a new cluster
      3. Cluster installation may fail due to image-registry cluster operator never going healthy, or the installer not being able to generate a storage account key for the bootstrap node to fetch its ignition config.  
      

      Actual results:

      storage.AccountsClient#ListAccountSAS: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="TooManyRequests" Message="The request is being throttled as the limit has been reached for operation type - Read_ObservationWindow_00:05:00. For more information, see - https://aka.ms/srpthrottlinglimits"
      

      Expected results:

      Cluster installs successfully
      

      Additional info:

      Raising this as a bug since this issue will be persistent across all cluster installations should one exceed the threshold.  It will also impact the image-registry pod health.  
      

      Attachments

        Issue Links

          Activity

            People

              fmissi Flavian Missi
              bvesel@redhat.com Benjamin Vesel
              wen wang wen wang
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: