Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59386

[release-4.19] GCP scaling is slow when scaling large volumes of nodes

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • Done
    • Bug Fix
    • Hide
      Cause: Each request to reconcile the state of a Machine happens sequentially
      Consequence: When scaling large numbers of nodes, scaling can be slow as it take time for each Machine to run through several reconciles given the depth of the queue is large
      Fix: Enable concurrent reconciliation, up to 10 Machines at a time
      Result: The speed at which Machines are processed in scale events is more performant
      Show
      Cause: Each request to reconcile the state of a Machine happens sequentially Consequence: When scaling large numbers of nodes, scaling can be slow as it take time for each Machine to run through several reconciles given the depth of the queue is large Fix: Enable concurrent reconciliation, up to 10 Machines at a time Result: The speed at which Machines are processed in scale events is more performant
    • None
    • None
    • None
    • None

      Description of problem:

          In a recent hackathon with Amadeus, we found that scaling of Nodes on GCP (0-400) was bottlenecked by sequential processing of reconcile requests in the Machine API provider for GCP.

Adding the ability to configure and then scale the nodes using parallel execution of 10 reconciles at once, significantly improved the performance.

      Version-Release number of selected component (if applicable):

      How reproducible:

      Steps to Reproduce:

          1.
    2.
    3.

      Actual results:

      Expected results:

      Additional info:

              joelspeed Joel Speed
              joelspeed Joel Speed
              None
              None
              Meha Bhalodiya Meha Bhalodiya
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: