Details
-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.16.0
-
No
-
Rejected
-
False
Description
Description of problem:
TRT tooling has detected a persistent spike in the disruption to connections to the internal registry of 4.16, aws, sdn, micro upgrade jobs (affecting new and reused connections). The disruption appears to be specific to this configuration. The spike begins around January 3, 2024.
Version-Release number of selected component (if applicable):
4.16
How reproducible:
P95 has been relatively consistent since January 3rd.
Expected results:
Disruption should be at or below 4.15 in the same configuration.
Additional info:
- The same version/environment, running OVN, does not show this spike. - The same version, on GCP, with SDN does not show this spike.
Example jobs showing the disruption:
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade/1743954248183320576
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade/1743670072548790272
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade/1743376264250527744
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade/1743041268742623232
- https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-sdn-upgrade/1742377557749141504
Here is a link to grafana showing the spike & more failing jobs (under "Last 500 Job Results within Window): https://grafana-loki.ci.openshift.org/d/ISnBj4LVk/disruption?var-platform=aws&var-percentile=P95&var-releases=4.15&var-releases=4.16&var-upgrade_type=micro&var-networks=sdn&var-networks=ovn&var-topologies=ha&var-architectures=amd64&var-lookback=1&orgId=1&var-min_job_runs=10&var-master_node_updated=Y&var-min_disruption_regression=0&var-min_disruption_job_list=4.5&var-master_nodes_updated=Y&var-min_relevance=0&from=1702160883639&to=1704752883639&var-backend=image-registry-new-connections&var-backend=image-registry-reused-connections&var-backend=ci-cluster-network-liveness-new-connections&var-backend=ci-cluster-network-liveness-reused-connections
