Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13128

Several SNO clusters fail to install because the OLM operator is unavailable/degraded (Both operator-lifecycle-manager and operator-lifecycle-manager-packageserver)

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • 5/9: telco reviewed
    • None
    • Rejected
    • Chaos, Doomsday, Err
    • 3
    • Done
    • Bug Fix
    • Hide
      * Before this update, Operator Lifecycle Manager (OLM) could cause failed installations due to initialization errors when the API server was busy. This update fixes the issue by adding a one minute retry interval for initialization errors.
      (link:https://issues.redhat.com/browse/OCPBUGS-13128[*OCPBUGS-13128*])
      Show
      * Before this update, Operator Lifecycle Manager (OLM) could cause failed installations due to initialization errors when the API server was busy. This update fixes the issue by adding a one minute retry interval for initialization errors. (link: https://issues.redhat.com/browse/OCPBUGS-13128 [* OCPBUGS-13128 *])
    • None
    • None
    • None
    • None

      Description of problem:

      While deploy 3671 SNOs via ACM and ZTP, 19 SNO clusters failed to install because the clusterversion object complained that the cluster operator operator-lifecycle-manager is not available.

      Version-Release number of selected component (if applicable):

      Hub OCP 4.12.14
SNO Deployed OCP 4.13.0-rc.6
ACM - 2.8.0-DOWNSTREAM-2023-04-30-18-44-29

      How reproducible:

      19 out of 51 failed clusters out of 3671 total installs
~.5% of installs might experience this however it represents ~37% of all install failures

      Steps to Reproduce:

      1.
2.
3.

      Actual results:

      # cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get clusterversion --no-headers"
vm00096 version         False   True   15h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available                                 
vm00334 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available                                 
vm00593 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available                                 
vm01095 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available                                 
vm01192 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available                                 
vm01447 version         False   True   18h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01566 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01707 version         False   True   17h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01742 version         False   True   15h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01798 version         False   True   13h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01810 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02020 version         False   True   19h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02091 version         False   True   20h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02363 version         False   True   13h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02590 version         False   True   20h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02908 version         False   True   18h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03253 version         False   True   14h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03500 version         False   True   17h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03654 version         False   True   17h   Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available

      Expected results:

       

      Additional info:

      There appears to be two distinguishing failure signatures in the list of cluster operators, every cluster shows that the OLM isn't available and is degraded and more than half of the clusters show no information regarding the operator-lifecycle-manager-packageserver.

      # cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get co operator-lifecycle-manager --no-headers"
vm00096 operator-lifecycle-manager         False   True   True   15h   
vm00334 operator-lifecycle-manager         False   True   True   19h   
vm00593 operator-lifecycle-manager         False   True   True   19h   
vm01095 operator-lifecycle-manager         False   True   True   19h   
vm01192 operator-lifecycle-manager         False   True   True   19h   
vm01447 operator-lifecycle-manager         False   True   True   18h   
vm01566 operator-lifecycle-manager         False   True   True   19h   
vm01707 operator-lifecycle-manager         False   True   True   17h   
vm01742 operator-lifecycle-manager         False   True   True   15h   
vm01798 operator-lifecycle-manager         False   True   True   13h   
vm01810 operator-lifecycle-manager         False   True   True   19h   
vm02020 operator-lifecycle-manager         False   True   True   19h   
vm02091 operator-lifecycle-manager         False   True   True   20h   
vm02363 operator-lifecycle-manager         False   True   True   13h   
vm02590 operator-lifecycle-manager         False   True   True   20h   
vm02908 operator-lifecycle-manager         False   True   True   18h   
vm03253 operator-lifecycle-manager         False   True   True   14h   
vm03500 operator-lifecycle-manager         False   True   True   17h   
vm03654 operator-lifecycle-manager         False   True   True   17h
# cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get co operator-lifecycle-manager-packageserver --no-headers"
vm00096 operator-lifecycle-manager-packageserver                                 
vm00334 operator-lifecycle-manager-packageserver         False   True   False   19h   
vm00593 operator-lifecycle-manager-packageserver         False   True   False   19h   
vm01095 operator-lifecycle-manager-packageserver                                 
vm01192 operator-lifecycle-manager-packageserver                                 
vm01447 operator-lifecycle-manager-packageserver                                 
vm01566 operator-lifecycle-manager-packageserver         False   True   False   19h   
vm01707 operator-lifecycle-manager-packageserver                                 
vm01742 operator-lifecycle-manager-packageserver         False   True   False   15h   
vm01798 operator-lifecycle-manager-packageserver                                 
vm01810 operator-lifecycle-manager-packageserver                                 
vm02020 operator-lifecycle-manager-packageserver                                 
vm02091 operator-lifecycle-manager-packageserver         False   True   False   20h   
vm02363 operator-lifecycle-manager-packageserver         False   True   False   13h   
vm02590 operator-lifecycle-manager-packageserver         False   True   False   20h   
vm02908 operator-lifecycle-manager-packageserver         False   True   False   18h   
vm03253 operator-lifecycle-manager-packageserver                                 
vm03500 operator-lifecycle-manager-packageserver                                 
vm03654 operator-lifecycle-manager-packageserver

      Viewing the pods in the  openshift-operator-lifecycle-manager for these clusters shows no packageserver pod:

      # cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get po -n openshift-operator-lifecycle-manager"
vm00096
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-9rm9j         1/1     Running     1 (15h ago)   15h
collect-profiles-28053720-kbsdn          0/1     Completed   0             33m
collect-profiles-28053735-dzkf8          0/1     Completed   0             18m
collect-profiles-28053750-skvcn          0/1     Completed   0             3m1s
olm-operator-66658fffbb-gj294            1/1     Running     0             15h
package-server-manager-654759688-bxnwj   1/1     Running     0             15h
vm00334
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-xcw9r         1/1     Running     1 (19h ago)   19h
collect-profiles-28053720-ppq6x          0/1     Completed   0             32m
collect-profiles-28053735-r2rvw          0/1     Completed   0             18m
collect-profiles-28053750-lgb4r          0/1     Completed   0             3m2s
olm-operator-66658fffbb-t4nxg            1/1     Running     0             19h
package-server-manager-654759688-6n7gp   1/1     Running     0             19h
vm00593
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-rwfwp         1/1     Running     1 (19h ago)   19h
collect-profiles-28053720-7p6tq          0/1     Completed   0             33m
collect-profiles-28053735-nqzn9          0/1     Completed   0             18m
collect-profiles-28053750-zppm6          0/1     Completed   0             3m2s
olm-operator-66658fffbb-4gcpv            1/1     Running     0             19h
package-server-manager-654759688-rbjdw   1/1     Running     0             19h
vm01095
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-2tp6j         1/1     Running     0          19h
collect-profiles-28053720-bnrfz          0/1     Completed   0          33m
collect-profiles-28053735-p8bl5          0/1     Completed   0          18m
collect-profiles-28053750-mg9nv          0/1     Completed   0          3m2s
olm-operator-66658fffbb-cb95l            1/1     Running     0          19h
package-server-manager-654759688-2mqdm   1/1     Running     0          19h
vm01192
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-2crgg         1/1     Running     0          19h
collect-profiles-28053720-2rknm          0/1     Completed   0          33m
collect-profiles-28053735-wc5dn          0/1     Completed   0          18m
collect-profiles-28053750-g5bhj          0/1     Completed   0          3m2s
olm-operator-66658fffbb-5hlh4            1/1     Running     0          19h
package-server-manager-654759688-xfp24   1/1     Running     0          19h
vm01447
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-p8gd4         1/1     Running     0             18h
collect-profiles-28053720-kjw4w          0/1     Completed   0             33m
collect-profiles-28053735-k7xxp          0/1     Completed   0             17m
collect-profiles-28053750-fn5gq          0/1     Completed   0             3m3s
olm-operator-66658fffbb-rshjq            1/1     Running     1 (18h ago)   18h
package-server-manager-654759688-hrmfd   1/1     Running     0             18h
vm01566
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-gbrnj         1/1     Running     0             19h
collect-profiles-28053720-2wdcp          0/1     Completed   0             33m
collect-profiles-28053735-t7x5b          0/1     Completed   0             18m
collect-profiles-28053750-wdmtt          0/1     Completed   0             3m3s
olm-operator-66658fffbb-fsxrx            1/1     Running     0             19h
package-server-manager-654759688-4mdz8   1/1     Running     1 (19h ago)   19h
vm01707
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-f2ns6         1/1     Running     0          17h
collect-profiles-28053720-72sjt          0/1     Completed   0          33m
collect-profiles-28053735-qzgx4          0/1     Completed   0          18m
collect-profiles-28053750-mrpbl          0/1     Completed   0          3m3s
olm-operator-66658fffbb-jwp2l            1/1     Running     0          17h
package-server-manager-654759688-f7bm4   1/1     Running     0          17h
vm01742
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-lhv6f         1/1     Running     1 (15h ago)   15h
collect-profiles-28053720-4kqtf          0/1     Completed   0             33m
collect-profiles-28053735-hw7kp          0/1     Completed   0             18m
collect-profiles-28053750-6ztq2          0/1     Completed   0             3m4s
olm-operator-66658fffbb-5sqlc            1/1     Running     0             15h
package-server-manager-654759688-n6sms   1/1     Running     0             15h
vm01798
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-kx7nx         1/1     Running     2 (13h ago)   13h
collect-profiles-28053720-7vlqq          0/1     Completed   0             33m
collect-profiles-28053735-m8ltn          0/1     Completed   0             18m
collect-profiles-28053750-hrfnk          0/1     Completed   0             3m4s
olm-operator-66658fffbb-5z74m            1/1     Running     1 (13h ago)   13h
package-server-manager-654759688-6jbnz   1/1     Running     0             13h
vm01810
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-v5vr6         1/1     Running     2 (19h ago)   19h
collect-profiles-28053720-m26dn          0/1     Completed   0             33m
collect-profiles-28053735-64j7f          0/1     Completed   0             18m
collect-profiles-28053750-qf69b          0/1     Completed   0             3m4s
olm-operator-66658fffbb-gxt2b            1/1     Running     0             19h
package-server-manager-654759688-dz6p6   1/1     Running     0             19h
vm02020
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-2qqk6         1/1     Running     0          19h
collect-profiles-28053720-5cktx          0/1     Completed   0          33m
collect-profiles-28053735-ls6n9          0/1     Completed   0          18m
collect-profiles-28053750-bj6gl          0/1     Completed   0          3m4s
olm-operator-66658fffbb-zsr4g            1/1     Running     0          19h
package-server-manager-654759688-2dnfd   1/1     Running     0          19h
vm02091
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-whftg         1/1     Running     1 (20h ago)   20h
collect-profiles-28053720-zqcbs          0/1     Completed   0             33m
collect-profiles-28053735-v8lf5          0/1     Completed   0             18m
collect-profiles-28053750-rshdd          0/1     Completed   0             3m5s
olm-operator-66658fffbb-876ps            1/1     Running     0             20h
package-server-manager-654759688-smc8q   1/1     Running     0             20h
vm02363
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-zgn5m         1/1     Running     1 (13h ago)   13h
collect-profiles-28053720-dpkqq          0/1     Completed   0             33m
collect-profiles-28053735-nfqmf          0/1     Completed   0             18m
collect-profiles-28053750-jfhdz          0/1     Completed   0             3m5s
olm-operator-66658fffbb-bbrgb            1/1     Running     1 (13h ago)   13h
package-server-manager-654759688-7pv96   1/1     Running     0             13h
vm02590
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-v9mvc         1/1     Running     2 (20h ago)   20h
collect-profiles-28053720-pfcbd          0/1     Completed   0             33m
collect-profiles-28053735-5dxbl          0/1     Completed   0             18m
collect-profiles-28053750-95f6g          0/1     Completed   0             3m5s
olm-operator-66658fffbb-5knlj            1/1     Running     0             20h
package-server-manager-654759688-7qkgb   1/1     Running     0             20h
vm02908
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-cnmjf         1/1     Running     0             18h
collect-profiles-28053720-ks6h7          0/1     Completed   0             33m
collect-profiles-28053735-r682b          0/1     Completed   0             18m
collect-profiles-28053750-9jrx4          0/1     Completed   0             3m5s
olm-operator-66658fffbb-7bd2v            1/1     Running     1 (18h ago)   18h
package-server-manager-654759688-5r6gq   1/1     Running     0             18h
vm03253
NAME                                     READY   STATUS      RESTARTS      AGE
catalog-operator-94b8bfddc-8wtgg         1/1     Running     2 (14h ago)   14h
collect-profiles-28053720-kwcgk          0/1     Completed   0             33m
collect-profiles-28053735-dv5hx          0/1     Completed   0             18m
collect-profiles-28053750-8xbmw          0/1     Completed   0             3m6s
olm-operator-66658fffbb-f2n9f            1/1     Running     0             14h
package-server-manager-654759688-tjlc9   1/1     Running     0             14h
vm03500
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-wdq9b         1/1     Running     0          17h
collect-profiles-28053720-jcmwf          0/1     Completed   0          33m
collect-profiles-28053735-tjw5j          0/1     Completed   0          18m
collect-profiles-28053750-5mjq9          0/1     Completed   0          3m6s
olm-operator-66658fffbb-q92bg            1/1     Running     0          17h
package-server-manager-654759688-2z656   1/1     Running     0          17h
vm03654
NAME                                     READY   STATUS      RESTARTS   AGE
catalog-operator-94b8bfddc-vq9wt         1/1     Running     0          17h
collect-profiles-28053720-dlknz          0/1     Completed   0          33m
collect-profiles-28053735-mshs7          0/1     Completed   0          18m
collect-profiles-28053750-86xrc          0/1     Completed   0          3m6s
olm-operator-66658fffbb-5qd99            1/1     Running     0          17h

       

       

        1. must-gather-OLMDegraded-vm02363.tar.gz
          22.32 MB
          Alex Krzos
        2. must-gather-OLMDegraded-vm01798.tar.gz
          22.57 MB
          Alex Krzos
        3. must-gather-OLMDegraded-vm01742.tar.gz
          24.00 MB
          Alex Krzos
        4. must-gather-OLMDegraded-vm00096.tar.gz
          24.13 MB
          Alex Krzos
        5. must-gather-OLMDegraded-vm03253.tar.gz
          25.31 MB
          Alex Krzos
        6. must-gather-OLMDegraded-vm03500.tar.gz
          26.77 MB
          Alex Krzos
        7. must-gather-OLMDegraded-vm02020.tar.gz
          29.73 MB
          Alex Krzos
        8. must-gather-OLMDegraded-vm01447.tar.gz
          30.31 MB
          Alex Krzos
        9. must-gather-OLMDegraded-vm02091.tar.gz
          29.84 MB
          Alex Krzos
        10. must-gather-OLMDegraded-vm01810.tar.gz
          30.03 MB
          Alex Krzos
        11. must-gather-OLMDegraded-vm00593.tar.gz
          30.64 MB
          Alex Krzos
        12. must-gather-OLMDegraded-vm00334.tar.gz
          30.37 MB
          Alex Krzos
        13. must-gather-OLMDegraded-vm01095.tar.gz
          31.35 MB
          Alex Krzos
        14. must-gather-OLMDegraded-vm01707.tar.gz
          31.89 MB
          Alex Krzos
        15. must-gather-OLMDegraded-vm03654.tar.gz
          31.97 MB
          Alex Krzos
        16. must-gather-OLMDegraded-vm01192.tar.gz
          33.61 MB
          Alex Krzos
        17. must-gather-OLMDegraded-vm01566.tar.gz
          33.44 MB
          Alex Krzos
        18. must-gather-OLMDegraded-vm02908.tar.gz
          33.85 MB
          Alex Krzos
        19. must-gather-OLMDegraded-vm02590.tar.gz
          35.47 MB
          Alex Krzos

              tshort@redhat.com Todd Short
              akrzos@redhat.com Alex Krzos
              None
              None
              Kui Wang Kui Wang
              Michael Peter Michael Peter
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: