Details
-
Story
-
Resolution: Unresolved
-
Normal
-
None
-
4.12
-
False
-
None
-
False
-
Impediment
-
Workloads Sprint 225, Workloads Sprint 226, Workloads Sprint 227, Workloads Sprint 228, Workloads Sprint 229, Workloads Sprint 230, Workloads Sprint 231, Workloads Sprint 232, Workloads Sprint 233, Workloads Sprint 234, Workloads Sprint 235, Workloads Sprint 236, Workloads Sprint 237, Workloads Sprint 238, Workloads Sprint 239, Workloads Sprint 240, Workloads Sprint 241
Description
In TRT-529, we observed readiness probes taking longer than 1 second, and threfore failing, on openshift-config-operator in the 4.12 nightly payloads which started happening around 2022-08-29 (started at 06:44:46 UTC).
The symptom is that the pod was ready but we see that it was intermittently failing readiness probes.
You can observe the behavior in this chart from this prow job.
In the chart, add "openshift-config-operator" in the RegEx box to focus on openshift-config-operator. The green lines represent the pod being ready; the red lines underneath the green show the readiness probes failing intermittently.
This issue is happening with other pods but we focused only on openshift-config-operator for this Jira.
We also opened up TRT-566 to make a separate test to help track the problem and TRT-567 to increase log verbosity so we can trace what is happening.
The tracing showed more logs but it's hard to tell where the bottleneck is. Here's a sample log (with more verbosity) where the problem is happening in this prow job.
