Pods on the cluster were unable to join the CNI with errors like:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-0_ocm-production-25f68u541bb9kgsrjurt7plkgtprs9k9-training_1ac2dd93-0382-4cb3-ab2f-ed57c6685fcc_0(1c7392c7570418d47e30f441e81a06cb527e3ef027b1a69eb0faa66f6cabd7fe): error adding pod ocm-production-25f68u541bb9kgsrjurt7plkgtprs9k9-training_etcd-0 to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [ocm-production-25f68u541bb9kgsrjurt7plkgtprs9k9-training/etcd-0/1ac2dd93-0382-4cb3-ab2f-ed57c6685fcc:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[ocm-production-25f68u541bb9kgsrjurt7plkgtprs9k9-training/etcd-0 1c7392c7570418d47e30f441e81a06cb527e3ef027b1a69eb0faa66f6cabd7fe] [ocm-production-25f68u541bb9kgsrjurt7plkgtprs9k9-training/etcd-0 1c7392c7570418d47e30f441e81a06cb527e3ef027b1a69eb0faa66f6cabd7fe] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded

and it was resolved by performing an OVN Database Reset/Rebuild https://access.redhat.com/articles/6963671#rebuild_ovn_on_ocp_4_8_4_13

4.12.27

It has happened many times, but we are not sure how to reproduce it

After performing the steps in https://access.redhat.com/articles/6963671#rebuild_ovn_on_ocp_4_8_4_13, the cluster recovered on its own over the span of 15 minutes.

In some sense that performing a manual OVN DB Rebuild isn't needed, but also that this causes the network clusteroperator to report that it is degraded