In TRT-529, we observed readiness probes taking longer than 1 second, and threfore failing, on openshift-config-operator in the 4.12 nightly payloads which started happening around 2022-08-29 (started at 06:44:46 UTC).

The symptom is that the pod was ready but we see that it was intermittently failing readiness probes.

You can observe the behavior in this chart from this prow job.

In the chart, add "openshift-config-operator" in the RegEx box to focus on openshift-config-operator. The green lines represent the pod being ready; the red lines underneath the green show the readiness probes failing intermittently.

This issue is happening with other pods but we focused only on openshift-config-operator for this Jira.

We also opened up TRT-566 to make a separate test to help track the problem and TRT-567 to increase log verbosity so we can trace what is happening.

The tracing showed more logs but it's hard to tell where the bottleneck is. Here's a sample log (with more verbosity) where the problem is happening in this prow job.