Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12881

Many SNOs failed to complete install because many cluster operators are in degraded or unavailable state. (4.13.0-rc.5)

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Low
    • No
    • None
    • None
    • None
    • OCPEDGE Sprint 253, OCPEDGE Sprint 254, OCPEDGE Sprint 255
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      While attempting to install 2430 SNOs from ACM via ZTP using the assisted-installer, 13 clusters failed with many cluster operaters either in degraded or unavailable state.  It is unclear if these are all the same failure or separate failures.

      Version-Release number of selected component (if applicable):

      Hub OCP - 4.12.14
      SNO OCP - 4.13.0-rc.5
      ACM - 2.8.0-DOWNSTREAM-2023-04-25-22-18-13

      How reproducible:

      Rare, 13 out of 213 sno install failures where 213 SNO installs failed out of 2430 attempts.

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

       

      Additional info:

      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00068   Ready    control-plane,master,worker   18h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          18h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00139   Ready    control-plane,master   17h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          17h     Working towards 4.13.0-rc.5: 518 of 841 done (61% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00386   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00535   Ready    control-plane,master   17h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          17h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00671   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00699   Ready    control-plane,master   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Working towards 4.13.0-rc.5: 481 of 841 done (57% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00887   Ready    control-plane,master,worker   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01117   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01146   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01798   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm02034   Ready    control-plane,master   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Working towards 4.13.0-rc.5: 518 of 841 done (61% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm02385   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm02424   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available

        1. must-gather-ManyOperatorsIncomplete-vm00068.tar.gz
          0.3 kB
          Alex Krzos
        2. must-gather-ManyOperatorsIncomplete-vm00585.tar.gz
          2 kB
          Alex Krzos
        3. must-gather-ManyOperatorsIncomplete-vm01146.tar.gz
          0.3 kB
          Alex Krzos
        4. must-gather-ManyOperatorsIncomplete-vm00887.tar.gz
          0.3 kB
          Alex Krzos
        5. must-gather-ManyOperatorsIncomplete-vm01117.tar.gz
          2 kB
          Alex Krzos
        6. must-gather-ManyOperatorsIncomplete-vm00386.tar.gz
          2 kB
          Alex Krzos
        7. must-gather-ManyOperatorsIncomplete-vm02385.tar.gz
          0.3 kB
          Alex Krzos
        8. must-gather-ManyOperatorsIncomplete-vm00139.tar.gz
          6.72 MB
          Alex Krzos
        9. must-gather-ManyOperatorsIncomplete-vm02034.tar.gz
          7.60 MB
          Alex Krzos
        10. must-gather-ManyOperatorsIncomplete-vm00699.tar.gz
          9.30 MB
          Alex Krzos
        11. must-gather-ManyOperatorsIncomplete-vm02424.tar.gz
          9.46 MB
          Alex Krzos
        12. must-gather-ManyOperatorsIncomplete-vm00671.tar.gz
          10.39 MB
          Alex Krzos
        13. must-gather-ManyOperatorsIncomplete-vm01798.tar.gz
          11.33 MB
          Alex Krzos
        14. sosreport-vm00699-2023-04-27-bpgpfiy.tar.xz
          21.65 MB
          Alex Krzos
        15. sosreport-vm00139-2023-04-27-zlklsyg.tar.xz
          22.52 MB
          Alex Krzos
        16. sosreport-vm02034-2023-04-27-zvaeryd.tar.xz
          23.88 MB
          Alex Krzos
        17. sosreport-vm00535-2023-04-27-xmebomv.tar.xz
          24.54 MB
          Alex Krzos
        18. sosreport-vm00671-2023-04-27-wectbjw.tar.xz
          31.56 MB
          Alex Krzos
        19. sosreport-vm01798-2023-04-27-ajnvdnq.tar.xz
          34.62 MB
          Alex Krzos
        20. sosreport-vm01117-2023-04-27-uywoivd.tar.xz
          35.75 MB
          Alex Krzos
        21. sosreport-vm00887-2023-04-27-dchxdes.tar.xz
          38.83 MB
          Alex Krzos
        22. sosreport-vm00386-2023-04-27-fagperb.tar.xz
          40.18 MB
          Alex Krzos
        23. sosreport-vm01146-2023-04-27-oqxivpr.tar.xz
          40.96 MB
          Alex Krzos
        24. sosreport-vm00068-2023-04-27-agjnika.tar.xz
          43.63 MB
          Alex Krzos
        25. sosreport-vm02424-2023-04-27-mnzdein.tar.xz
          35.83 MB
          Alex Krzos
        26. sosreport-vm02385-2023-04-27-rnqwtyw.tar.xz
          41.19 MB
          Alex Krzos

              bzamalut@redhat.com Bulat Zamalutdinov
              akrzos@redhat.com Alex Krzos
              None
              None
              Pedro Jose Amoedo Martinez Pedro Jose Amoedo Martinez
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: