While upgrading 1241 SNOs using TALM, 3 clusters were unable to upgrade because their etcd operator was degraded.

Deployed SNO OCP 4.10.32
Attempted to upgrade to 4.11.5

Rare because only 3 out of 1241 clusters had this issue
Out of all the upgrade failures, it was 3 out 26 failures

1.
2.
3.

[root@e27-h01-000-r650 common-and-group]# oc --kubeconfig=/root/hv-vm/sno/manifests/sno00423/kubeconfig get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.32   True        False         41h     Error while reconciling 4.10.32: the cluster operator etcd is degraded
[root@e27-h01-000-r650 common-and-group]# oc --kubeconfig=/root/hv-vm/sno/manifests/sno00454/kubeconfig get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.32   True        False         41h     Error while reconciling 4.10.32: the cluster operator etcd is degraded
[root@e27-h01-000-r650 common-and-group]# oc --kubeconfig=/root/hv-vm/sno/manifests/sno01049/kubeconfig get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.32   True        False         39h     Error while reconciling 4.10.32: the cluster operator etcd is degraded

SNO clusters to not degrade etcd operator and prevent upgrading

  - lastTransitionTime: "2022-10-26T04:36:26Z"                                                                                                                                                
    message: |-                                                                                                                                                                               
      EtcdEndpointsDegraded: no etcd members are present                                                                                                                                      
      UpgradeBackupControllerDegraded: etcdmembers.etcd.operator.openshift.io "sno01049" not found                                                                                            
    reason: EtcdEndpoints_ErrorUpdatingEtcdEndpoints::UpgradeBackupController_Error                                                                                                           
    status: "True"                                                                                                                                                                            
    type: Degraded 

[root@sno00423 ~]# netstat -ntlp | egrep "2380|2379|9978"
tcp6       0      0 :::2379                 :::*                    LISTEN      9047/etcd           
tcp6       0      0 :::2380                 :::*                    LISTEN      9047/etcd           
tcp6       0      0 :::9978                 :::*                    LISTEN      9047/etcd