-
Bug
-
Resolution: Can't Do
-
Undefined
-
None
-
4.15.z, 4.16.z
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
Yes
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
etcd requests slow, while running etcd pod disruption chaos test on ARO
"alertstate": "pending",
"namespace": "openshift-etcd",
"severity": "critical"
Version-Release number of selected component (if applicable):
4.15.35, 4.16.30
How reproducible:
4.15.35 - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/63721/rehearse-63721-periodic-ci-redhat-chaos-prow-scripts-main-4.19-nightly-krkn-hub-tests-aro/1923007716721168384
4.16.30 - https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/63721/rehearse-63721-periodic-ci-redhat-chaos-prow-scripts-main-4.16-nightly-krkn-hub-tests-aro/1925550329978425344
Steps to Reproduce:
1.Trigger ARO job on PR : https://github.com/openshift/release/pull/63721
2.
3.
Actual results:
ft-etcd",
"severity": "critical"
},
{
"alertname": "etcdGRPCRequestsSlow",
"alertstate": "pending",
"namespace": "openshift-etcd",
"severity": "critical"
},
{
"alertname": "etcdGRPCRequestsSlow",
"alertstate": "pending",
"namespace": "openshift-etcd",
"severity": "critical"
}
],
"scenario": "pod_disruption_scenarios",
"run_id": "3ec3a8c7-f3a5-46b7-a15a-c9ca66ade57e"
}
}
2025-05-22 15:32:06,604 [INFO] telemetry data will be stored on s3 bucket folder: https://ulnmf9xv7j.execute-api.us-west-2.amazonaws.com/production/files/prow-ocp-aro-4.16/1747927498-3ec3a8c7-f3a5-46b7-a15a-c9ca66ade57e-etcd-pod-disruption
2025-05-22 15:32:06,604 [INFO] telemetry upload log: /tmp/1747927498-3ec3a8c7-f3a5-46b7-a15a-c9ca66ade57e-etcd-pod-disruption.log
2025-05-22 15:32:18,208 [INFO] Alerts checking is enabled
2025-05-22 15:32:18,257 [WARNING] 10 minutes avg. 99th etcd fsync latency on etcd-ci-op-msrkbh8x-60263-t5gkp-master-1 higher than 10ms. 0.010798285028016361s
2025-05-22 15:32:18,377 [WARNING] 10 minutes avg. 99th etcd commit latency on etcd-ci-op-msrkbh8x-60263-t5gkp-master-2 higher than 30ms. 0.10985270913187688s
2025-05-22 15:32:18,597 [CRITICAL] etcd grpc requests are slow.
2025-05-22 15:32:19,608 [INFO] Capturing metrics using file /home/krkn/kraken/config/metrics-report.yaml
2025-05-22 15:32:22,067 [ERROR] Critical alerts are firing, please check; exiting
Expected results:
It should pass
Additional info:
