Uploaded image for project: 'Red Hat Advanced Cluster Management'
  1. Red Hat Advanced Cluster Management
  2. ACM-20933

Large delay in Image Based Installed clusters becoming managed during scale test

XMLWordPrintable

    • 3
    • False
    • Hide

      None

      Show
      None
    • False
    • Release Note Not Required
    • Important
    • None

      Description of problem:

      While installing 3672 SNOs via Image based install operator there is a very noticeable delay in the time post a cluster is installed and when the cluster becomes managed as shown in the results graph. (Review graph with annotations showing delays) The delays appear to have began when 2384 clusters were managed, lasted about 20 minutes before clusters started becoming managed again. The next delay was about 30 minutes and you can see further delays. This behavior is much different than agent based cluster installs in which no large 20 or 30 minute plus delays prevent any newly installed clusters from becoming managed. 

      Version-Release number of selected component (if applicable):

      Hub OCP 4.17.3

      Deployed cluster OCP 4.17.3

      ACM 2.12 - 2.12.0-DOWNSTREAM-2024-10-25-20-49-39

      How reproducible:

      This has appeared in all 6 ibi large scale runs so far. so 100% reproducible in the scale environment

      Steps to Reproduce:

      1.  
      2.  
      3. ...

      Actual results:

      Expected results:

      Additional info:

        1. 0.log
          45.70 MB
          Ting Xue
        2. 0.log.20241029-230829.gz
          4.42 MB
          Ting Xue
        3. 0.log.20241030-005942.gz
          4.47 MB
          Ting Xue
        4. 0.log.20241030-024945.gz
          4.45 MB
          Ting Xue
        5. 0.log.20241030-070041
          50.02 MB
          Ting Xue
        6. 218data.jpg
          192 kB
          Ting Xue
        7. 220data.csv
          615 kB
          Ting Xue
        8. IBI-Run32-managedcluster-delay-share2-20241029-111958.png
          108 kB
          Ting Xue
        9. Screenshot from 2024-10-30 09-36-59.png
          73 kB
          Ting Xue
        10. Screenshot from 2024-10-30 09-37-33.png
          63 kB
          Ting Xue
        11. hub-acm-must-gather.tar.gz
          113.02 MB
          Ting Xue
        12. ici.yaml
          3 kB
          Ting Xue
        13. image (1).png
          401 kB
          Ting Xue
        14. image (2).png
          914 kB
          Ting Xue
        15. managedcluster-import-controller-v2-b94cf9cb4-gpb9j.log.gz
          1.64 MB
          Ting Xue
        16. managedcluster.yaml
          13 kB
          Ting Xue
        17. manifestworks.yaml
          386 kB
          Ting Xue
        18. monitor_data.csv
          35 kB
          Ting Xue
        19. result3k1209.csv
          793 kB
          Ting Xue
        20. share2-20241023-172114.png
          111 kB
          Ting Xue
        21. share2-20241024-172348.png
          118 kB
          Ting Xue
        22. share2-20241030-040658.png
          126 kB
          Ting Xue
        23. hive-machinepool-0.log
          2 kB
          Ting Xue
        24. hiveadmission-7bf55c8dfc-xmfj5.log
          3.40 MB
          Ting Xue
        25. hiveadmission-7bf55c8dfc-p86gg.log
          3.48 MB
          Ting Xue
        26. hive-machinepool-0-1.log
          2 kB
          Ting Xue
        27. hiveadmission-7bf55c8dfc-p86gg-1.log
          3.48 MB
          Ting Xue
        28. hive-controllers.log
          10.45 MB
          Ting Xue
        29. hive-clustersync-0.log
          14.87 MB
          Ting Xue
        30. hive-operator-6b5b4d874f-zp4v5.log
          15.60 MB
          Ting Xue
        31. hive-controllers-1.log
          10.45 MB
          Ting Xue
        32. hive-operator-6b5b4d874f-zp4v5-1.log
          15.60 MB
          Ting Xue
        33. hive-clustersync-0-1.log
          14.87 MB
          Ting Xue
        34. Run10-with-hiveconfig.png
          116 kB
          Ting Xue
        35. hive-machinepool-0-machinepool.log
          2 kB
          Ting Xue
        36. hive-operator-7f54fd7c6c-h2h2t-hive-operator.log
          257 kB
          Ting Xue
        37. hive-operator-7f54fd7c6c-h2h2t-hive-operator-1.log
          257 kB
          Ting Xue
        38. hive-machinepool-0-machinepool-1.log
          2 kB
          Ting Xue
        39. hiveadmission-649c597d74-dwfqs-hiveadmission.log
          5.26 MB
          Ting Xue
        40. hiveadmission-649c597d74-h25f9-hiveadmission.log
          5.25 MB
          Ting Xue
        41. hive-controllers-74d74bf7fc-jvl7t-manager.log
          6.00 MB
          Ting Xue
        42. hiveadmission-649c597d74-dwfqs-hiveadmission-1.log
          5.26 MB
          Ting Xue
        43. hiveadmission-649c597d74-h25f9-hiveadmission-1.log
          5.25 MB
          Ting Xue
        44. hive-clustersync-0-clustersync.log
          39.49 MB
          Ting Xue
        45. hive-clustersync-0-clustersync-1.log
          39.49 MB
          Ting Xue

              efried.openshift Eric Fried
              rhn-support-txue Ting Xue
              Ting Xue Ting Xue
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated: