Observed errors during upgrade - Overloaded Network, No CNI configuration file & request took too long & upgrade fails.
Job Link: 
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-s390x/1709689971566186496
 
Observing errors during upgrade and its observed on OCT 6 build : 1710052283481329664 & 1710173084788461568 , OCT 4 1709569171760615424 1709448375402762240
 
1. dropped internal Raft message since sending buffer is full (overloaded network)
2. reason/NetworkNotReady network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? (18 times)
3. container/etcd src/podLog apply request took too long
 
Note:
NO CNI configuration file error started from this build (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-s390x/1690921297640427520) post that we got only once success build (https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-multiarch-master-nightly-4.14-upgrade-from-nightly-4.13-ocp-ovn-remote-libvirt-s390x/1692008592795766784) .
 
Log lines for reference.
 
time="2023-10-05T00:43:30Z" level=info msg="added interval: Oct 05 00:36:20.721 - 1s W ns/openshift-etcd pod/etcd-libvirt-s390x-1-0-0ef-s7zrs-master-1 node/libvirt-s390x-1-0-0ef-s7zrs-master-1 uid/362773fc-6e95-4e30-ac07-3ca2e4d02433 container/etcd src/podLog dropped internal Raft message since sending buffer is full (overloaded network)" pod=etcd-libvirt-s390x-1-0-0ef-s7zrs-master-1
 
processed event: {TypeMeta:

ObjectMeta:{Name:dns-default-d4sk2.178b0d925ab5c167 GenerateName: Namespace:openshift-dns SelfLink: UID:0f22aa68-e64c-424e-8c7e-aced6a2699d1 ResourceVersion:106648 Generation:0 CreationTimestamp:2023-10-05 00:10:06 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[monitor.openshift.io/observed-recreation-count: monitor.openshift.io/observed-update-count:1] OwnerReferences:[] Finalizers:[] ManagedFields:[{Manager:kubelet Operation:Update APIVersion:v1 Time:2023-10-05 00:10:09 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:count":{},"f:firstTimestamp":{},"f:involvedObject":{},"f:lastTimestamp":{},"f:message":{},"f:reason":{},"f:reportingComponent":{},"f:reportingInstance":{},"f:source":\{"f:component":{},"f:host":{}},"f:type":{}} Subresource:}]} InvolvedObject:{Kind:Pod Namespace:openshift-dns Name:dns-default-d4sk2 UID:247d347a-feab-4283-8470-89d2878240d5 APIVersion:v1 ResourceVersion:106395 FieldPath:} Reason:NetworkNotReady Message:network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? Source:{Component:kubelet Host:libvirt-s390x-1-0-0ef-s7zrs-worker-0-hbsm2} FirstTimestamp:2023-10-05 00:10:06 +0000 UTC LastTimestamp:2023-10-05 00:10:09 +0000 UTC Count:2 Type:Warning EventTime:0001-01-01 00:00:00 +0000 UTC Series:nil Action: Related:nil ReportingController:kubelet ReportingInstance:libvirt-s390x-1-0-0ef-s7zrs-worker-0-hbsm2}
resulting new interval: reason/NetworkNotReady network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started? (2 times) from: 2023-10-05 00:10:09 +0000 UTC to 2023-10-05 00:10:10 +0000 UTC

time="2023-10-05T00:43:18Z" level=info msg="fetching logs between 2023-10-04T22:43:40Z and 2023-10-05T00:43:17Z" pod=etcd-libvirt-s390x-1-0-0ef-s7zrs-master-0
time="2023-10-05T00:43:18Z" level=info msg="added interval: Oct 05 00:29:55.483 - 1s W ns/openshift-etcd pod/etcd-libvirt-s390x-1-0-0ef-s7zrs-master-0 node/libvirt-s390x-1-0-0ef-s7zrs-master-0 uid/12656c65-ad77-4a65-a44e-6653e12622b6 container/etcd src/podLog apply request took too long" pod=etcd-libvirt-s390x-1-0-0ef-s7zrs-master-0