-
Bug
-
Resolution: Unresolved
-
Undefined
-
4.13, 4.14, 4.15, 4.17, 4.18, 4.19, 4.20
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
In Progress
-
Release Note Not Required
-
None
-
None
-
None
-
None
-
None
Description of problem
CI is flaky because of test failures such as the following:
[sig-devex] check registry.redhat.io is available and samples operator can import sample imagestreams [It] run sample related validations [apigroup:config.openshift.io][apigroup:image.openshift.io] [Skipped:Disconnected] [Suite:openshift/conformance/parallel]
github.com/openshift/origin/test/extended/tbr_health/check.go:14
[FAILED] Expected
<string>: failed to get imagestream: imagestreams.image.openshift.io "nodejs" not found
to be empty
In [It] at: github.com/openshift/origin/test/extended/tbr_health/check.go:18 @ 05/13/25 18:43:32.476
This particular failure comes from https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_cluster-ingress-operator/1232/pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-ovn-single-node/1922322816472977408. Search.ci has other similar failures.
Version-Release number of selected component (if applicable)
I have seen this in CI jobs for 4.13 through 4.20.
How reproducible
Presently, search.ci shows the following stats for the past two days:
periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ovn-single-node-rt-upgrade (all) - 1 runs, 100% failed, 100% of failures match = 100% impact pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-ovn-single-node (all) - 9 runs, 44% failed, 25% of failures match = 11% impact periodic-ci-openshift-release-master-ci-4.19-e2e-aws-upgrade-ovn-single-node (all) - 30 runs, 53% failed, 6% of failures match = 3% impact periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 30 runs, 43% failed, 15% of failures match = 7% impact periodic-ci-openshift-multiarch-master-nightly-4.17-ocp-e2e-azure-ovn-multi-a-a (all) - 4 runs, 25% failed, 100% of failures match = 25% impact periodic-ci-openshift-multiarch-master-nightly-4.18-ocp-e2e-upgrade-azure-ovn-multi-a-a (all) - 3 runs, 33% failed, 100% of failures match = 33% impact
The search times out if I try to search more than two days without further limiting the scope. Many, but not all, of the failures are on single-node jobs, so I tried limiting the scope to single-node jobs. Then I can extend the search scope to 14 days, and then I get the following stats:
periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ovn-single-node-rt-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-ovn-single-node (all) - 13 runs, 54% failed, 14% of failures match = 8% impact periodic-ci-openshift-release-master-ci-4.19-e2e-aws-upgrade-ovn-single-node (all) - 360 runs, 33% failed, 2% of failures match = 1% impact periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 370 runs, 39% failed, 3% of failures match = 1% impact aggregated-aws-ovn-single-node-upgrade-4.19-micro-release-openshift-release-analysis-aggregator (all) - 42 runs, 21% failed, 67% of failures match = 14% impact periodic-ci-openshift-release-master-ci-4.18-e2e-aws-upgrade-ovn-single-node (all) - 43 runs, 40% failed, 6% of failures match = 2% impact openshift-ovn-kubernetes-2561-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 10 runs, 80% failed, 13% of failures match = 10% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-single-node-techpreview (all) - 37 runs, 32% failed, 8% of failures match = 3% impact aggregator-periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 19 runs, 74% failed, 7% of failures match = 5% impact periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-aws-upgrade-ovn-single-node (all) - 36 runs, 50% failed, 6% of failures match = 3% impact periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-aws-upgrade-ovn-single-node (all) - 2 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-single-node (all) - 46 runs, 22% failed, 10% of failures match = 2% impact periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-ovn-single-node (all) - 10 runs, 40% failed, 25% of failures match = 10% impact periodic-ci-openshift-release-master-nightly-4.18-aws-ovn-network-flow-matrix-single-node (all) - 8 runs, 38% failed, 33% of failures match = 13% impact periodic-ci-openshift-release-master-nightly-4.20-upgrade-from-stable-4.19-e2e-aws-upgrade-ovn-single-node (all) - 37 runs, 35% failed, 8% of failures match = 3% impact pull-ci-openshift-console-operator-main-e2e-aws-ovn-single-node (all) - 6 runs, 67% failed, 25% of failures match = 17% impact periodic-ci-openshift-release-master-nightly-4.20-upgrade-from-stable-4.19-e2e-aws-upgrade-ovn-single-node-network-flow-matrix (all) - 8 runs, 100% failed, 13% of failures match = 13% impact periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-single-node (all) - 41 runs, 46% failed, 5% of failures match = 2% impact periodic-ci-openshift-release-master-nightly-4.17-e2e-metal-ovn-single-node-rt-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact pull-ci-openshift-cluster-ingress-operator-master-e2e-aws-ovn-single-node (all) - 13 runs, 54% failed, 14% of failures match = 8% impact periodic-ci-openshift-release-master-ci-4.19-e2e-aws-upgrade-ovn-single-node (all) - 360 runs, 33% failed, 2% of failures match = 1% impact periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 370 runs, 39% failed, 3% of failures match = 1% impact aggregated-aws-ovn-single-node-upgrade-4.19-micro-release-openshift-release-analysis-aggregator (all) - 42 runs, 21% failed, 67% of failures match = 14% impact periodic-ci-openshift-release-master-ci-4.18-e2e-aws-upgrade-ovn-single-node (all) - 43 runs, 40% failed, 6% of failures match = 2% impact openshift-ovn-kubernetes-2561-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 10 runs, 80% failed, 13% of failures match = 10% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-single-node-techpreview (all) - 37 runs, 32% failed, 8% of failures match = 3% impact aggregator-periodic-ci-openshift-release-master-ci-4.20-e2e-aws-upgrade-ovn-single-node (all) - 19 runs, 74% failed, 7% of failures match = 5% impact periodic-ci-openshift-release-master-nightly-4.19-upgrade-from-stable-4.18-e2e-aws-upgrade-ovn-single-node (all) - 36 runs, 50% failed, 6% of failures match = 3% impact periodic-ci-openshift-release-master-nightly-4.15-upgrade-from-stable-4.14-e2e-aws-upgrade-ovn-single-node (all) - 2 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-single-node (all) - 46 runs, 22% failed, 10% of failures match = 2% impact periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-ovn-single-node (all) - 10 runs, 40% failed, 25% of failures match = 10% impact periodic-ci-openshift-release-master-nightly-4.18-aws-ovn-network-flow-matrix-single-node (all) - 8 runs, 38% failed, 33% of failures match = 13% impact periodic-ci-openshift-release-master-nightly-4.20-upgrade-from-stable-4.19-e2e-aws-upgrade-ovn-single-node (all) - 37 runs, 35% failed, 8% of failures match = 3% impact pull-ci-openshift-console-operator-main-e2e-aws-ovn-single-node (all) - 6 runs, 67% failed, 25% of failures match = 17% impact periodic-ci-openshift-release-master-nightly-4.20-upgrade-from-stable-4.19-e2e-aws-upgrade-ovn-single-node-network-flow-matrix (all) - 8 runs, 100% failed, 13% of failures match = 13% impact periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-single-node (all) - 41 runs, 46% failed, 5% of failures match = 2% impact
Steps to Reproduce
1. Post a PR and have bad luck.
2. Check search.ci: https://search.dptools.openshift.org/?search=check+registry.redhat.io+is+available+and+samples+operator+can+import+sample+imagestreams+run+sample+related+validations&maxAge=48h&context=2&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Actual results
CI fails.
Expected results
CI passes, or fails on some other test failure.
Additional info
All of the failures are for "nodejs". However, this is probably only because "nodejs" is the first imagestream that the test checks: https://github.com/openshift/origin/blob/6e15b2cb23fe0d14d5fb4ed3cb8fcaa59aec5f76/test/extended/util/framework.go#L313-L341
