-
Bug
-
Resolution: Cannot Reproduce
-
Undefined
-
None
-
4.18, 4.19
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
Observing the below etcd testfailure in 4.18 and 4.19 PowerVS runs
1. [sig-etcd] etcd should not log excessive took too long messages
{ Etcd logged 10493 'took too long' messages, this test fails on any value over 10000 as this is a strong indicator that etcd was very unhealthy throughout the run. This can cause sparodic e2e failures and disruption and typically indicates faster disks are needed. These log message intervals are included in spyglass chart artifacts and can be used to correlate with disruption and failed tests.}
2. [bz-etcd][invariant] alert/etcdMemberCommunicationSlow should not be at or above info
{ etcdMemberCommunicationSlow was at or above info for at least 2m22s on platformidentification.JobType{Release:"4.19", FromRelease:"", Platform:"", Architecture:"ppc64le", Network:"ovn", Topology:"ha"} (maxAllowed=0s): pending for 9m30s, firing for 2m22s: Apr 10 00:51:38.876 - 58s W namespace/openshift-etcd node/192.168.124.10:9979 pod/etcd-p-lon06-1-capi-419-7mdjj-master-0 alert/etcdMemberCommunicationSlow alertstate/firing severity/warning ALERTS{To="890f003e4491e681", alertname="etcdMemberCommunicationSlow", alertstate="firing", endpoint="etcd-metrics", instance="192.168.124.10:9979", job="etcd", namespace="openshift-etcd", pod="etcd-p-lon06-1-capi-419-7mdjj-master-0", prometheus="openshift-monitoring/k8s", service="etcd", severity="warning"} Apr 10 00:52:08.876 - 28s W namespace/openshift-etcd node/192.168.124.11:9979 pod/etcd-p-lon06-1-capi-419-7mdjj-master-1 alert/etcdMemberCommunicationSlow alertstate/firing severity/warning ALERTS{To="28d2c243d792bfce", alertname="etcdMemberCommunicationSlow", alertstate="firing", endpoint="etcd-metrics", instance="192.168.124.11:9979", job="etcd", namespace="openshift-etcd", pod="etcd-p-lon06-1-capi-419-7mdjj-master-1", prometheus="openshift-monitoring/k8s", service="etcd", severity="warning"} Apr 10 00:52:08.876 - 28s W namespace/openshift-etcd node/192.168.124.12:9979 pod/etcd-p-lon06-1-capi-419-7mdjj-master-2 alert/etcdMemberCommunicationSlow alertstate/firing severity/warning ALERTS{To="890f003e4491e681", alertname="etcdMemberCommunicationSlow", alertstate="firing", endpoint="etcd-metrics", instance="192.168.124.12:9979", job="etcd", namespace="openshift-etcd", pod="etcd-p-lon06-1-capi-419-7mdjj-master-2", prometheus="openshift-monitoring/k8s", service="etcd", severity="warning"} Apr 10 00:52:08.876 - 28s W namespace/openshift-etcd node/192.168.124.11:9979 pod/etcd-p-lon06-1-capi-419-7mdjj-master-1 alert/etcdMemberCommunicationSlow alertstate/firing severity/warning ALERTS{To="dbe386c0426da004", alertname="etcdMemberCommunicationSlow", alertstate="firing", endpoint="etcd-metrics", instance="192.168.124.11:9979", job="etcd", namespace="openshift-etcd", pod="etcd-p-lon06-1-capi-419-7mdjj-master-1", prometheus="openshift-monitoring/k8s", service="etcd", severity="warning"}}
