Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12881

Many SNOs failed to complete install because many cluster operators are in degraded or unavailable state. (4.13.0-rc.5)

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 3
    • Low
    • No
    • None
    • None
    • None
    • OCPEDGE Sprint 253, OCPEDGE Sprint 254, OCPEDGE Sprint 255
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      While attempting to install 2430 SNOs from ACM via ZTP using the assisted-installer, 13 clusters failed with many cluster operaters either in degraded or unavailable state.  It is unclear if these are all the same failure or separate failures.

      Version-Release number of selected component (if applicable):

      Hub OCP - 4.12.14
      SNO OCP - 4.13.0-rc.5
      ACM - 2.8.0-DOWNSTREAM-2023-04-25-22-18-13

      How reproducible:

      Rare, 13 out of 213 sno install failures where 213 SNO installs failed out of 2430 attempts.

      Steps to Reproduce:

      1.
      2.
      3.
      

      Actual results:

       

      Expected results:

       

      Additional info:

      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00068   Ready    control-plane,master,worker   18h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          18h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00139   Ready    control-plane,master   17h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          17h     Working towards 4.13.0-rc.5: 518 of 841 done (61% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00386   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00535   Ready    control-plane,master   17h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          17h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00671   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: an unknown error has occurred: MultipleErrors
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm00699   Ready    control-plane,master   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Working towards 4.13.0-rc.5: 481 of 841 done (57% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm00887   Ready    control-plane,master,worker   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01117   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01146   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm01798   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                  AGE   VERSION
      node/vm02034   Ready    control-plane,master   19h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          19h     Working towards 4.13.0-rc.5: 518 of 841 done (61% complete)
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm02385   Ready    control-plane,master,worker   20h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          20h     Unable to apply 4.13.0-rc.5: some cluster operators are not available
      NAME           STATUS   ROLES                         AGE   VERSION
      node/vm02424   Ready    control-plane,master,worker   16h   v1.26.3+379cd9f
      
      NAME                                         VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      clusterversion.config.openshift.io/version             False       True          16h     Unable to apply 4.13.0-rc.5: some cluster operators are not available

        1. sosreport-vm02424-2023-04-27-mnzdein.tar.xz
          35.83 MB
        2. sosreport-vm02385-2023-04-27-rnqwtyw.tar.xz
          41.19 MB
        3. sosreport-vm02034-2023-04-27-zvaeryd.tar.xz
          23.88 MB
        4. sosreport-vm01798-2023-04-27-ajnvdnq.tar.xz
          34.62 MB
        5. sosreport-vm01146-2023-04-27-oqxivpr.tar.xz
          40.96 MB
        6. sosreport-vm01117-2023-04-27-uywoivd.tar.xz
          35.75 MB
        7. sosreport-vm00887-2023-04-27-dchxdes.tar.xz
          38.83 MB
        8. sosreport-vm00699-2023-04-27-bpgpfiy.tar.xz
          21.65 MB
        9. sosreport-vm00671-2023-04-27-wectbjw.tar.xz
          31.56 MB
        10. sosreport-vm00535-2023-04-27-xmebomv.tar.xz
          24.54 MB
        11. sosreport-vm00386-2023-04-27-fagperb.tar.xz
          40.18 MB
        12. sosreport-vm00139-2023-04-27-zlklsyg.tar.xz
          22.52 MB
        13. sosreport-vm00068-2023-04-27-agjnika.tar.xz
          43.63 MB
        14. must-gather-ManyOperatorsIncomplete-vm02424.tar.gz
          9.46 MB
        15. must-gather-ManyOperatorsIncomplete-vm02385.tar.gz
          0.3 kB
        16. must-gather-ManyOperatorsIncomplete-vm02034.tar.gz
          7.60 MB
        17. must-gather-ManyOperatorsIncomplete-vm01798.tar.gz
          11.33 MB
        18. must-gather-ManyOperatorsIncomplete-vm01146.tar.gz
          0.3 kB
        19. must-gather-ManyOperatorsIncomplete-vm01117.tar.gz
          2 kB
        20. must-gather-ManyOperatorsIncomplete-vm00887.tar.gz
          0.3 kB
        21. must-gather-ManyOperatorsIncomplete-vm00699.tar.gz
          9.30 MB
        22. must-gather-ManyOperatorsIncomplete-vm00671.tar.gz
          10.39 MB
        23. must-gather-ManyOperatorsIncomplete-vm00585.tar.gz
          2 kB
        24. must-gather-ManyOperatorsIncomplete-vm00386.tar.gz
          2 kB
        25. must-gather-ManyOperatorsIncomplete-vm00139.tar.gz
          6.72 MB
        26. must-gather-ManyOperatorsIncomplete-vm00068.tar.gz
          0.3 kB

              bzamalut@redhat.com Bulat Zamalutdinov
              akrzos@redhat.com Alex Krzos
              None
              None
              Pedro Jose Amoedo Martinez Pedro Jose Amoedo Martinez
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: