Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-402

Reduce number of dashboard pods to reduce RHODS min requirements

    XMLWordPrintable

Details

    • Story
    • Resolution: Unresolved
    • Normal
    • None
    • None
    • Dashboard, Platform

    Description

      As part of our transition to KFNBC and to pass performance tests, we increased the number of dashboard pods to 5.  After several performance enhancements, the load on dashboard usage has gone down significantly.  I propose we drop the numer of pods back down to 3, which should be a very small 1 line change.   It's more important now that we're adding more controllers for serving.

      Here are the results from kpouget2 's test with 3 dashboard pods quoted from slack:

      Test configuration:
          300 users starting with a delay of 2 seconds
      RHODS 1.18.0-8 running on OCP v4.10.15
          Total of 65 nodes in the cluster
          3 m6i.2xlarge master nodes
          2 m6a.xlarge infra nodes
          50 m6i.2xlarge rhods_compute nodes
          10 m6i.2xlarge test_pods_only nodes
          Test pods running on another cluster
      300/300 users succeeded 
      looks all good :) (edited) 

      Attachments

        1. image (5).png
          image (5).png
          151 kB
        2. newplot(231).png
          newplot(231).png
          310 kB
        3. screenshot-1.png
          screenshot-1.png
          154 kB
        4. screenshot-2.png
          screenshot-2.png
          117 kB
        5. screenshot-3.png
          screenshot-3.png
          146 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              cchase@redhat.com Chris Chase
              RHOAI Platform
              Votes:
              11 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:

                PagerDuty