Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-15246

Large delay in Image Based Installed clusters becoming managed during scale test

XMLWordPrintable

    • 3
    • False
    • None
    • False
    • SF Train-20, SF Train-21, SF Train-22, SF Train-23
    • Important
    • None

      Description of problem:

      While installing 3672 SNOs via Image based install operator there is a very noticeable delay in the time post a cluster is installed and when the cluster becomes managed as shown in the results graph. (Review graph with annotations showing delays) The delays appear to have began when 2384 clusters were managed, lasted about 20 minutes before clusters started becoming managed again. The next delay was about 30 minutes and you can see further delays. This behavior is much different than agent based cluster installs in which no large 20 or 30 minute plus delays prevent any newly installed clusters from becoming managed. 

      Version-Release number of selected component (if applicable):

      Hub OCP 4.17.3

      Deployed cluster OCP 4.17.3

      ACM 2.12 - 2.12.0-DOWNSTREAM-2024-10-25-20-49-39

      How reproducible:

      This has appeared in all 6 ibi large scale runs so far. so 100% reproducible in the scale environment

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

        1. 0.log
          45.70 MB
        2. 0.log.20241029-230829.gz
          4.42 MB
        3. 0.log.20241030-005942.gz
          4.47 MB
        4. 0.log.20241030-024945.gz
          4.45 MB
        5. 0.log.20241030-070041
          50.02 MB
        6. 218data.jpg
          218data.jpg
          192 kB
        7. hub-acm-must-gather.tar.gz
          113.02 MB
        8. IBI-Run32-managedcluster-delay-share2-20241029-111958.png
          IBI-Run32-managedcluster-delay-share2-20241029-111958.png
          108 kB
        9. ici.yaml
          3 kB
        10. image (1).png
          image (1).png
          401 kB
        11. image (2).png
          image (2).png
          914 kB
        12. managedcluster.yaml
          13 kB
        13. managedcluster-import-controller-v2-b94cf9cb4-gpb9j.log.gz
          1.64 MB
        14. manifestworks.yaml
          386 kB
        15. monitor_data.csv
          35 kB
        16. result3k1209.csv
          793 kB
        17. Screenshot from 2024-10-30 09-36-59.png
          Screenshot from 2024-10-30 09-36-59.png
          73 kB
        18. Screenshot from 2024-10-30 09-37-33.png
          Screenshot from 2024-10-30 09-37-33.png
          63 kB
        19. share2-20241023-172114.png
          share2-20241023-172114.png
          111 kB
        20. share2-20241024-172348.png
          share2-20241024-172348.png
          118 kB
        21. share2-20241030-040658.png
          share2-20241030-040658.png
          126 kB

              zyin@redhat.com Zhiwei Yin
              akrzos@redhat.com Alex Krzos
              Hui Chen Hui Chen
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: