Details
-
Bug
-
Resolution: Done-Errata
-
Undefined
-
4.13
-
No
-
Chaos, Doomsday, Err
-
3
-
Rejected
-
False
-
-
-
Bug Fix
-
Done
-
5/9: telco reviewed
Description
Description of problem:
While deploy 3671 SNOs via ACM and ZTP, 19 SNO clusters failed to install because the clusterversion object complained that the cluster operator operator-lifecycle-manager is not available.
Version-Release number of selected component (if applicable):
Hub OCP 4.12.14 SNO Deployed OCP 4.13.0-rc.6 ACM - 2.8.0-DOWNSTREAM-2023-04-30-18-44-29
How reproducible:
19 out of 51 failed clusters out of 3671 total installs ~.5% of installs might experience this however it represents ~37% of all install failures
Steps to Reproduce:
1. 2. 3.
Actual results:
# cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get clusterversion --no-headers"
vm00096 version False True 15h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm00334 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm00593 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01095 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01192 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01447 version False True 18h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01566 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01707 version False True 17h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01742 version False True 15h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01798 version False True 13h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm01810 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02020 version False True 19h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02091 version False True 20h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02363 version False True 13h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02590 version False True 20h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm02908 version False True 18h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03253 version False True 14h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03500 version False True 17h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
vm03654 version False True 17h Unable to apply 4.13.0-rc.6: the cluster operator operator-lifecycle-manager is not available
Expected results:
Additional info:
There appears to be two distinguishing failure signatures in the list of cluster operators, every cluster shows that the OLM isn't available and is degraded and more than half of the clusters show no information regarding the operator-lifecycle-manager-packageserver.
# cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get co operator-lifecycle-manager --no-headers"
vm00096 operator-lifecycle-manager False True True 15h
vm00334 operator-lifecycle-manager False True True 19h
vm00593 operator-lifecycle-manager False True True 19h
vm01095 operator-lifecycle-manager False True True 19h
vm01192 operator-lifecycle-manager False True True 19h
vm01447 operator-lifecycle-manager False True True 18h
vm01566 operator-lifecycle-manager False True True 19h
vm01707 operator-lifecycle-manager False True True 17h
vm01742 operator-lifecycle-manager False True True 15h
vm01798 operator-lifecycle-manager False True True 13h
vm01810 operator-lifecycle-manager False True True 19h
vm02020 operator-lifecycle-manager False True True 19h
vm02091 operator-lifecycle-manager False True True 20h
vm02363 operator-lifecycle-manager False True True 13h
vm02590 operator-lifecycle-manager False True True 20h
vm02908 operator-lifecycle-manager False True True 18h
vm03253 operator-lifecycle-manager False True True 14h
vm03500 operator-lifecycle-manager False True True 17h
vm03654 operator-lifecycle-manager False True True 17h
# cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo -n '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get co operator-lifecycle-manager-packageserver --no-headers"
vm00096 operator-lifecycle-manager-packageserver
vm00334 operator-lifecycle-manager-packageserver False True False 19h
vm00593 operator-lifecycle-manager-packageserver False True False 19h
vm01095 operator-lifecycle-manager-packageserver
vm01192 operator-lifecycle-manager-packageserver
vm01447 operator-lifecycle-manager-packageserver
vm01566 operator-lifecycle-manager-packageserver False True False 19h
vm01707 operator-lifecycle-manager-packageserver
vm01742 operator-lifecycle-manager-packageserver False True False 15h
vm01798 operator-lifecycle-manager-packageserver
vm01810 operator-lifecycle-manager-packageserver
vm02020 operator-lifecycle-manager-packageserver
vm02091 operator-lifecycle-manager-packageserver False True False 20h
vm02363 operator-lifecycle-manager-packageserver False True False 13h
vm02590 operator-lifecycle-manager-packageserver False True False 20h
vm02908 operator-lifecycle-manager-packageserver False True False 18h
vm03253 operator-lifecycle-manager-packageserver
vm03500 operator-lifecycle-manager-packageserver
vm03654 operator-lifecycle-manager-packageserver
Viewing the pods in the openshift-operator-lifecycle-manager for these clusters shows no packageserver pod:
# cat cluster-install-failures | grep OLM | awk '{print $1}' | xargs -I % sh -c "echo '% '; oc --kubeconfig /root/hv-vm/kc/%/kubeconfig get po -n openshift-operator-lifecycle-manager"
vm00096
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-9rm9j 1/1 Running 1 (15h ago) 15h
collect-profiles-28053720-kbsdn 0/1 Completed 0 33m
collect-profiles-28053735-dzkf8 0/1 Completed 0 18m
collect-profiles-28053750-skvcn 0/1 Completed 0 3m1s
olm-operator-66658fffbb-gj294 1/1 Running 0 15h
package-server-manager-654759688-bxnwj 1/1 Running 0 15h
vm00334
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-xcw9r 1/1 Running 1 (19h ago) 19h
collect-profiles-28053720-ppq6x 0/1 Completed 0 32m
collect-profiles-28053735-r2rvw 0/1 Completed 0 18m
collect-profiles-28053750-lgb4r 0/1 Completed 0 3m2s
olm-operator-66658fffbb-t4nxg 1/1 Running 0 19h
package-server-manager-654759688-6n7gp 1/1 Running 0 19h
vm00593
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-rwfwp 1/1 Running 1 (19h ago) 19h
collect-profiles-28053720-7p6tq 0/1 Completed 0 33m
collect-profiles-28053735-nqzn9 0/1 Completed 0 18m
collect-profiles-28053750-zppm6 0/1 Completed 0 3m2s
olm-operator-66658fffbb-4gcpv 1/1 Running 0 19h
package-server-manager-654759688-rbjdw 1/1 Running 0 19h
vm01095
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-2tp6j 1/1 Running 0 19h
collect-profiles-28053720-bnrfz 0/1 Completed 0 33m
collect-profiles-28053735-p8bl5 0/1 Completed 0 18m
collect-profiles-28053750-mg9nv 0/1 Completed 0 3m2s
olm-operator-66658fffbb-cb95l 1/1 Running 0 19h
package-server-manager-654759688-2mqdm 1/1 Running 0 19h
vm01192
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-2crgg 1/1 Running 0 19h
collect-profiles-28053720-2rknm 0/1 Completed 0 33m
collect-profiles-28053735-wc5dn 0/1 Completed 0 18m
collect-profiles-28053750-g5bhj 0/1 Completed 0 3m2s
olm-operator-66658fffbb-5hlh4 1/1 Running 0 19h
package-server-manager-654759688-xfp24 1/1 Running 0 19h
vm01447
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-p8gd4 1/1 Running 0 18h
collect-profiles-28053720-kjw4w 0/1 Completed 0 33m
collect-profiles-28053735-k7xxp 0/1 Completed 0 17m
collect-profiles-28053750-fn5gq 0/1 Completed 0 3m3s
olm-operator-66658fffbb-rshjq 1/1 Running 1 (18h ago) 18h
package-server-manager-654759688-hrmfd 1/1 Running 0 18h
vm01566
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-gbrnj 1/1 Running 0 19h
collect-profiles-28053720-2wdcp 0/1 Completed 0 33m
collect-profiles-28053735-t7x5b 0/1 Completed 0 18m
collect-profiles-28053750-wdmtt 0/1 Completed 0 3m3s
olm-operator-66658fffbb-fsxrx 1/1 Running 0 19h
package-server-manager-654759688-4mdz8 1/1 Running 1 (19h ago) 19h
vm01707
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-f2ns6 1/1 Running 0 17h
collect-profiles-28053720-72sjt 0/1 Completed 0 33m
collect-profiles-28053735-qzgx4 0/1 Completed 0 18m
collect-profiles-28053750-mrpbl 0/1 Completed 0 3m3s
olm-operator-66658fffbb-jwp2l 1/1 Running 0 17h
package-server-manager-654759688-f7bm4 1/1 Running 0 17h
vm01742
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-lhv6f 1/1 Running 1 (15h ago) 15h
collect-profiles-28053720-4kqtf 0/1 Completed 0 33m
collect-profiles-28053735-hw7kp 0/1 Completed 0 18m
collect-profiles-28053750-6ztq2 0/1 Completed 0 3m4s
olm-operator-66658fffbb-5sqlc 1/1 Running 0 15h
package-server-manager-654759688-n6sms 1/1 Running 0 15h
vm01798
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-kx7nx 1/1 Running 2 (13h ago) 13h
collect-profiles-28053720-7vlqq 0/1 Completed 0 33m
collect-profiles-28053735-m8ltn 0/1 Completed 0 18m
collect-profiles-28053750-hrfnk 0/1 Completed 0 3m4s
olm-operator-66658fffbb-5z74m 1/1 Running 1 (13h ago) 13h
package-server-manager-654759688-6jbnz 1/1 Running 0 13h
vm01810
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-v5vr6 1/1 Running 2 (19h ago) 19h
collect-profiles-28053720-m26dn 0/1 Completed 0 33m
collect-profiles-28053735-64j7f 0/1 Completed 0 18m
collect-profiles-28053750-qf69b 0/1 Completed 0 3m4s
olm-operator-66658fffbb-gxt2b 1/1 Running 0 19h
package-server-manager-654759688-dz6p6 1/1 Running 0 19h
vm02020
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-2qqk6 1/1 Running 0 19h
collect-profiles-28053720-5cktx 0/1 Completed 0 33m
collect-profiles-28053735-ls6n9 0/1 Completed 0 18m
collect-profiles-28053750-bj6gl 0/1 Completed 0 3m4s
olm-operator-66658fffbb-zsr4g 1/1 Running 0 19h
package-server-manager-654759688-2dnfd 1/1 Running 0 19h
vm02091
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-whftg 1/1 Running 1 (20h ago) 20h
collect-profiles-28053720-zqcbs 0/1 Completed 0 33m
collect-profiles-28053735-v8lf5 0/1 Completed 0 18m
collect-profiles-28053750-rshdd 0/1 Completed 0 3m5s
olm-operator-66658fffbb-876ps 1/1 Running 0 20h
package-server-manager-654759688-smc8q 1/1 Running 0 20h
vm02363
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-zgn5m 1/1 Running 1 (13h ago) 13h
collect-profiles-28053720-dpkqq 0/1 Completed 0 33m
collect-profiles-28053735-nfqmf 0/1 Completed 0 18m
collect-profiles-28053750-jfhdz 0/1 Completed 0 3m5s
olm-operator-66658fffbb-bbrgb 1/1 Running 1 (13h ago) 13h
package-server-manager-654759688-7pv96 1/1 Running 0 13h
vm02590
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-v9mvc 1/1 Running 2 (20h ago) 20h
collect-profiles-28053720-pfcbd 0/1 Completed 0 33m
collect-profiles-28053735-5dxbl 0/1 Completed 0 18m
collect-profiles-28053750-95f6g 0/1 Completed 0 3m5s
olm-operator-66658fffbb-5knlj 1/1 Running 0 20h
package-server-manager-654759688-7qkgb 1/1 Running 0 20h
vm02908
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-cnmjf 1/1 Running 0 18h
collect-profiles-28053720-ks6h7 0/1 Completed 0 33m
collect-profiles-28053735-r682b 0/1 Completed 0 18m
collect-profiles-28053750-9jrx4 0/1 Completed 0 3m5s
olm-operator-66658fffbb-7bd2v 1/1 Running 1 (18h ago) 18h
package-server-manager-654759688-5r6gq 1/1 Running 0 18h
vm03253
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-8wtgg 1/1 Running 2 (14h ago) 14h
collect-profiles-28053720-kwcgk 0/1 Completed 0 33m
collect-profiles-28053735-dv5hx 0/1 Completed 0 18m
collect-profiles-28053750-8xbmw 0/1 Completed 0 3m6s
olm-operator-66658fffbb-f2n9f 1/1 Running 0 14h
package-server-manager-654759688-tjlc9 1/1 Running 0 14h
vm03500
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-wdq9b 1/1 Running 0 17h
collect-profiles-28053720-jcmwf 0/1 Completed 0 33m
collect-profiles-28053735-tjw5j 0/1 Completed 0 18m
collect-profiles-28053750-5mjq9 0/1 Completed 0 3m6s
olm-operator-66658fffbb-q92bg 1/1 Running 0 17h
package-server-manager-654759688-2z656 1/1 Running 0 17h
vm03654
NAME READY STATUS RESTARTS AGE
catalog-operator-94b8bfddc-vq9wt 1/1 Running 0 17h
collect-profiles-28053720-dlknz 0/1 Completed 0 33m
collect-profiles-28053735-mshs7 0/1 Completed 0 18m
collect-profiles-28053750-86xrc 0/1 Completed 0 3m6s
olm-operator-66658fffbb-5qd99 1/1 Running 0 17h
