Version:
Hub OCP - 4.9.23
ACM - 2.5.0-DOWNSTREAM-2022-02-21-19-58-55
SNO OCP - 4.10.0-rc.5

Platform:
libvirt -SNO is a libvirt VM

Please specify:
Assisted-installer via ZTP/SiteConfig

Deployed 102 SNO clusters, 100 succeeded, 2 failed. One failure has multiple cluster operators not in available state.

What did you expect to happen?
Cluster operators to complete installing

Anything else we need to know?

1. oc get managedcluster -A | grep -i Unknown
   sno00044 true Unknown 16h
   sno00073 true Unknown 16h

1. oc --kubeconfig=/root/hv-sno/manifests/sno00073/kubeconfig get no,clusterversion,co
   NAME STATUS ROLES AGE VERSION
   node/sno00073 Ready master 16h v1.23.3+bba255d

NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
clusterversion.config.openshift.io/version False True 16h Unable to apply 4.10.0-rc.5: some cluster operators have not yet rolled out

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
clusteroperator.config.openshift.io/authentication 4.10.0-rc.5 False False True 16h OAuthServerRouteEndpointAccessibleControllerAvailable: failed to retrieve route from cache: route.route.openshift.io "oauth-openshift" not found...
clusteroperator.config.openshift.io/baremetal
clusteroperator.config.openshift.io/cloud-controller-manager
clusteroperator.config.openshift.io/cloud-credential 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/cluster-autoscaler
clusteroperator.config.openshift.io/config-operator 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/console
clusteroperator.config.openshift.io/csi-snapshot-controller 4.10.0-rc.5 False False False 16h CSISnapshotWebhookControllerAvailable: Waiting for a validating webhook Deployment pod to start
clusteroperator.config.openshift.io/dns 4.10.0-rc.5 False False True 16h DNS "default" is unavailable.
clusteroperator.config.openshift.io/etcd 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/image-registry
clusteroperator.config.openshift.io/ingress False True True 16h The "default" ingress controller reports Available=False: IngressControllerUnavailable: One or more status conditions indicate unavailable: DeploymentAvailable=Unknown (DeploymentAvailabilityUnknown: The deployment has no Available status condition set)
clusteroperator.config.openshift.io/insights
clusteroperator.config.openshift.io/kube-apiserver 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/kube-controller-manager 4.10.0-rc.5 True True True 16h MissingStaticPodControllerDegraded: static pod lifecycle failure - static pod: "kube-controller-manager" in namespace: "openshift-kube-controller-manager" for revision: 9 on node: "sno00073" didn't show up, waited: 2m30s
clusteroperator.config.openshift.io/kube-scheduler 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/kube-storage-version-migrator 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/machine-api
clusteroperator.config.openshift.io/machine-approver 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/machine-config False True True 16h Cluster not available for []
clusteroperator.config.openshift.io/marketplace 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/monitoring False True True 16h Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
clusteroperator.config.openshift.io/network 4.10.0-rc.5 True True False 16h DaemonSet "openshift-multus/network-metrics-daemon" is waiting for other operators to become ready...
clusteroperator.config.openshift.io/node-tuning False True False 16h DaemonSet "tuned" has no available Pod(s)
clusteroperator.config.openshift.io/openshift-apiserver 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/openshift-controller-manager 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/openshift-samples
clusteroperator.config.openshift.io/operator-lifecycle-manager
clusteroperator.config.openshift.io/operator-lifecycle-manager-catalog
clusteroperator.config.openshift.io/operator-lifecycle-manager-packageserver
clusteroperator.config.openshift.io/service-ca 4.10.0-rc.5 True False False 16h
clusteroperator.config.openshift.io/storage

1. oc --kubeconfig=/root/hv-sno/manifests/sno00073/kubeconfig get po -A
   NAMESPACE NAME READY STATUS RESTARTS AGE
   assisted-installer assisted-installer-controller-tj9m6 0/1 Completed 0 16h
   openshift-apiserver-operator openshift-apiserver-operator-859fd9d65-pmmgh 1/1 Running 1 (16h ago) 16h
   openshift-apiserver apiserver-76fbc95475-2drkw 2/2 Running 0 16h
   openshift-authentication-operator authentication-operator-576599b67c-6p6g7 1/1 Running 1 (16h ago) 16h
   openshift-cloud-credential-operator cloud-credential-operator-674b767f6d-r7gcq 2/2 Running 0 16h
   openshift-cluster-machine-approver machine-approver-858b998ffd-cckmx 2/2 Running 0 16h
   openshift-cluster-node-tuning-operator cluster-node-tuning-operator-859b44456f-vgcc4 1/1 Running 0 16h
   openshift-cluster-storage-operator csi-snapshot-controller-868bc48b4c-757df 1/1 Running 0 16h
   openshift-cluster-storage-operator csi-snapshot-controller-operator-56ff5dc584-54zdh 1/1 Running 0 16h
   openshift-cluster-storage-operator csi-snapshot-webhook-84f4c4d7bc-f6pnq 1/1 Running 0 16h
   openshift-cluster-version cluster-version-operator-654c9b8694-glbjb 1/1 Running 0 16h
   openshift-config-operator openshift-config-operator-649b659f-k669k 1/1 Running 1 (16h ago) 16h
   openshift-controller-manager-operator openshift-controller-manager-operator-55fcb5468c-c8ct2 1/1 Running 1 (16h ago) 16h
   openshift-controller-manager controller-manager-nnxtt 1/1 Running 0 16h
   openshift-dns-operator dns-operator-5745549f4-zndrt 2/2 Running 0 16h
   openshift-etcd-operator etcd-operator-6f95c76858-dq6jw 1/1 Running 1 (16h ago) 16h
   openshift-etcd etcd-sno00073 4/4 Running 0 16h
   openshift-etcd installer-2-sno00073 0/1 Completed 0 16h
   openshift-etcd installer-3-sno00073 0/1 Completed 0 16h
   openshift-ingress-operator ingress-operator-9cb76cfb6-tpnf5 2/2 Running 0 16h
   openshift-kube-apiserver-operator kube-apiserver-operator-79446fb849-n592b 1/1 Running 1 (16h ago) 16h
   openshift-kube-apiserver installer-2-retry-1-sno00073 0/1 Completed 0 16h
   openshift-kube-apiserver installer-2-sno00073 0/1 Error 0 16h
   openshift-kube-apiserver installer-3-sno00073 0/1 Completed 0 16h
   openshift-kube-apiserver installer-5-sno00073 0/1 Completed 0 16h
   openshift-kube-apiserver kube-apiserver-sno00073 5/5 Running 0 16h
   openshift-kube-controller-manager-operator kube-controller-manager-operator-5c4db569b7-4xwm2 1/1 Running 1 (16h ago) 16h
   openshift-kube-controller-manager installer-4-retry-1-sno00073 0/1 Completed 0 16h
   openshift-kube-controller-manager installer-4-sno00073 0/1 Error 0 16h
   openshift-kube-controller-manager installer-5-sno00073 0/1 Completed 0 16h
   openshift-kube-controller-manager installer-7-sno00073 0/1 Completed 0 16h
   openshift-kube-controller-manager installer-8-sno00073 0/1 Completed 0 16h
   openshift-kube-controller-manager installer-9-sno00073 0/1 Completed 0 16h
   openshift-kube-controller-manager kube-controller-manager-sno00073 4/4 Running 1 (16h ago) 16h
   openshift-kube-scheduler-operator openshift-kube-scheduler-operator-f99cd9549-5hd94 1/1 Running 1 (16h ago) 16h
   openshift-kube-scheduler installer-2-sno00073 0/1 Completed 0 16h
   openshift-kube-scheduler installer-3-sno00073 0/1 Completed 0 16h
   openshift-kube-scheduler installer-4-sno00073 0/1 Completed 0 16h
   openshift-kube-scheduler installer-5-sno00073 0/1 Completed 0 16h
   openshift-kube-scheduler installer-6-sno00073 0/1 Completed 0 16h
   openshift-kube-scheduler openshift-kube-scheduler-sno00073 3/3 Running 0 16h
   openshift-kube-scheduler revision-pruner-6-sno00073 0/1 Completed 0 16h
   openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator-5bbb478bd7-cspw6 1/1 Running 1 (16h ago) 16h
   openshift-kube-storage-version-migrator migrator-7b74cfb5dc-7j4sr 1/1 Running 0 16h
   openshift-machine-config-operator machine-config-daemon-ljw7v 2/2 Running 0 16h
   openshift-machine-config-operator machine-config-operator-7ddff4ccf8-t294b 1/1 Running 0 16h
   openshift-marketplace marketplace-operator-5d47b5d9f9-6nv8s 1/1 Running 0 16h
   openshift-monitoring cluster-monitoring-operator-59fc5b8f6c-bqbjc 2/2 Running 0 16h
   openshift-multus ip-reconciler-27437790-khz9h 0/1 Pending 0 16h
   openshift-multus multus-additional-cni-plugins-5fpc9 1/1 Running 0 16h
   openshift-multus multus-admission-controller-gf6rw 2/2 Running 0 16h
   openshift-multus multus-q7xhv 1/1 Running 0 16h
   openshift-multus network-metrics-daemon-lx5pk 2/2 Running 0 16h
   openshift-network-diagnostics network-check-source-865d4b5578-vg4tp 0/1 Pending 0 16h
   openshift-network-diagnostics network-check-target-q4hb2 1/1 Running 0 16h
   openshift-network-operator network-operator-6cddd54844-9ftm7 1/1 Running 0 16h
   openshift-oauth-apiserver apiserver-78f9b455db-ntrg9 1/1 Running 0 16h
   openshift-ovn-kubernetes ovnkube-master-qq4vr 6/6 Running 0 16h
   openshift-ovn-kubernetes ovnkube-node-f7wqh 5/5 Running 0 16h
   openshift-service-ca-operator service-ca-operator-847dfdd7fd-5fjmw 1/1 Running 1 (16h ago) 16h
   openshift-service-ca service-ca-65cf6d45b4-jjwnf 1/1 Running 0 16h