Uploaded image for project: 'Managed Service - Streams'
  1. Managed Service - Streams
  2. MGDSTRM-8740

Expose a configuration to read the max number of nodes for autoscaling configuration per instance type

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • None
    • MK - Sprint 220

      WHAT

      The Control Plane needs to know the maximum number of nodes configured for each supported instance so that it can create appropriate machine pools with the configuration needed by the autoscaler. This information has to be communicated to the fleetshard so that it can report back the maximum units that be handled in the configured nodes.

      The control plane will obtain this configuration via a dynamic scaling ConfigMap  (set via the Parameter in the service template). The ConfigMap needs to contain the max nodes limits for each instance type. There are some considerations to take:

      1. Consider the expansion of Kafka. It is quite possible that in the future we might want to allow customer to change the size of their kafka from x1 to x2 i.e expand their current kafka. The setting of this max value has to take this into account so that we do not set it to the max physcial OSD node limit (known value is 200 180)
      2. Consider OSD limit as indicated above

      Once the final value known for each machine pool, we need to store this in the database so that we can easily send it to the fleetshard. Also, any future change of the configuration won't afect already configured clusters (as no reconciliation of autoscaling configuration is planned for the moment)

       

      The configuration may look like

      ```

      maximum_node_limits:
        standard:
            limit: 50
        developer:
           limit: 30

      ```

      WHY

      This is need to:

      • configure the machine pool for each supported instance type
      • to communicate the value to the Fleetshard 

      DONE

      Include the following where applicable:

      • Configuration exposed 
      • Splitting logic applied when the cluster supports more instances (Staging)
      • The resulting value after the split is stored in the database for each supported instance type

              msorianod Miguel Soriano
              mchitimb-1 Manyanda Chitimbo
              MK - Control Plane
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: