-
Bug
-
Resolution: Unresolved
-
Normal
-
None
-
4.20.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Beginning around June 18th, we can see an increase in disruption metrics for 'kube-api-new-connections' in Metal and vSphere in '4.20' duing 'micro' upgrades. Changing the 'lookback days' to '1' allows us to see the pattern in the graph that leads to the 06/18/25 date. In '4.19' the pattern is less clear in Metal, but does appear present in VSphere. There are many occurrences of disruption within the job runs in the window of 4s and above. The percentage of non-zero disruption seems to be around 85-90% throughout the window.
The increased disruption values are present in a various different jobs, including the following:
- periodic-ci-openshift-release-master-ci-4.20-e2e-vsphere-runc-upgrade (example)
- periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-ovn-upgrade-runc (example)
- periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-ovn-bm-upgrade (example)
- periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-upgrade-ovn-ipv6 (example)
- It appears that the disruption occurs at the same time that there are alerts in the KubeletLog about failures to update lease.
- These intervals show that a portion of the disruption correlates with the kube-apiserver shutdown. It starts prior to the shutdown, but there still may be some relevance.
- Looking at PRs merged around 6/18, this one sticks out as potentially relevant (worth looking at, but I have no reason to believe it is the cause other than the timing).
- relates to
-
OCPBUGS-37153 kubenswrapper: Unexpected EOF during watch stream event decoding errors
-
- New
-