-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.12, 4.19, 4.20, 4.21
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem
CI is flaky because of test failures such as the following:
{ fail [github.com/openshift/origin/test/extended/project/project.go:217]: Expected
<bool>: true
to equal
<string>: e2e-test-project-api-cpnp5}
This particular failure comes from https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29900/pull-ci-openshift-origin-main-e2e-vsphere-ovn/1976388824435003392. Search.ci has other similar failures.
Version-Release number of selected component (if applicable)
I have seen this in 4.12, 4.19, 4.20, and 4.21 CI jobs.
How reproducible
Presently, search.ci shows the following stats for the past two days:
periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn-no-capabilities (all) - 1 runs, 100% failed, 100% of failures match = 100% impact openshift-kubernetes-2484-ci-4.21-e2e-aws-ovn-upgrade-out-of-change (all) - 4 runs, 75% failed, 33% of failures match = 25% impact pull-ci-openshift-cluster-monitoring-operator-main-okd-scos-e2e-aws-ovn (all) - 8 runs, 75% failed, 17% of failures match = 13% impact pull-ci-openshift-origin-main-e2e-aws-ovn-fips (all) - 60 runs, 32% failed, 11% of failures match = 3% impact openshift-kubernetes-2484-ci-4.21-e2e-azure-ovn-upgrade (all) - 48 runs, 79% failed, 3% of failures match = 2% impact periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade (all) - 80 runs, 38% failed, 40% of failures match = 15% impact periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-aws-ovn-upgrade (all) - 86 runs, 47% failed, 18% of failures match = 8% impact openshift-kubernetes-2484-ci-4.21-e2e-azure-ovn (all) - 4 runs, 25% failed, 100% of failures match = 25% impact openshift-kubernetes-2484-periodics-e2e-aws-ovn-conformance (all) - 50 runs, 32% failed, 63% of failures match = 20% impact periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-runc-upgrade (all) - 2 runs, 50% failed, 100% of failures match = 50% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-upgrade-fips (all) - 70 runs, 80% failed, 4% of failures match = 3% impact periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade (all) - 87 runs, 39% failed, 15% of failures match = 6% impact periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade (all) - 60 runs, 32% failed, 21% of failures match = 7% impact periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade (all) - 61 runs, 33% failed, 20% of failures match = 7% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-upgrade-fips (all) - 60 runs, 27% failed, 13% of failures match = 3% impact pull-ci-openshift-origin-main-e2e-vsphere-ovn (all) - 59 runs, 25% failed, 33% of failures match = 8% impact periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance (all) - 71 runs, 14% failed, 100% of failures match = 14% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-azure-ovn-multi-a-a (all) - 6 runs, 33% failed, 50% of failures match = 17% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-gcp-ovn-multi-x-ax (all) - 6 runs, 67% failed, 25% of failures match = 17% impact pull-ci-openshift-ovn-kubernetes-master-4.21-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade (all) - 27 runs, 63% failed, 6% of failures match = 4% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-vsphere-ovn-upi (all) - 6 runs, 50% failed, 67% of failures match = 33% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-vsphere-ovn (all) - 7 runs, 29% failed, 150% of failures match = 43% impact pull-ci-openshift-operator-framework-operator-controller-main-okd-scos-e2e-aws-ovn (all) - 30 runs, 50% failed, 7% of failures match = 3% impact periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn-conformance (all) - 62 runs, 10% failed, 233% of failures match = 23% impact pull-ci-openshift-ovn-kubernetes-release-4.20-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-rt-upgrade (all) - 2 runs, 50% failed, 100% of failures match = 50% impact rehearse-70150-pull-ci-openshift-multus-networkpolicy-main-okd-scos-e2e-aws-ovn (all) - 1 runs, 100% failed, 100% of failures match = 100% impact openshift-kubernetes-2484-ci-4.21-e2e-gcp-ovn-upgrade (all) - 64 runs, 31% failed, 5% of failures match = 2% impact periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade (all) - 86 runs, 28% failed, 25% of failures match = 7% impact openshift-kubernetes-2484-ci-4.21-upgrade-from-stable-4.20-e2e-gcp-ovn-rt-upgrade (all) - 50 runs, 22% failed, 9% of failures match = 2% impact openshift-kubernetes-2484-ci-4.21-upgrade-from-stable-4.20-e2e-azure-ovn-upgrade (all) - 69 runs, 77% failed, 4% of failures match = 3% impact periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-azure-aks-ovn-conformance (all) - 12 runs, 33% failed, 25% of failures match = 8% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-proxy (all) - 11 runs, 18% failed, 50% of failures match = 9% impact pull-ci-openshift-machine-config-operator-main-e2e-aws-ovn (all) - 5 runs, 20% failed, 100% of failures match = 20% impact pull-ci-openshift-ovn-kubernetes-release-4.20-e2e-azure-ovn-upgrade (all) - 6 runs, 33% failed, 100% of failures match = 33% impact pull-ci-openshift-origin-main-e2e-vsphere-ovn-upi (all) - 62 runs, 34% failed, 48% of failures match = 16% impact periodic-ci-openshift-release-master-ci-4.21-e2e-azure-ovn-upgrade (all) - 80 runs, 46% failed, 8% of failures match = 4% impact periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-gcp-ovn-rt-upgrade (all) - 71 runs, 28% failed, 25% of failures match = 7% impact periodic-ci-openshift-release-master-ci-4.12-e2e-azure-ovn-upgrade (all) - 11 runs, 55% failed, 17% of failures match = 9% impact periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-upgrade (all) - 85 runs, 33% failed, 21% of failures match = 7% impact pull-ci-openshift-origin-release-4.20-e2e-gcp-ovn (all) - 4 runs, 25% failed, 100% of failures match = 25% impact periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-azure-ovn-upgrade (all) - 81 runs, 56% failed, 16% of failures match = 9% impact periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-azure-ovn-multi-x-ax (all) - 5 runs, 80% failed, 50% of failures match = 40% impact pull-ci-openshift-origin-release-4.20-e2e-vsphere-ovn-upi (all) - 4 runs, 50% failed, 100% of failures match = 50% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-cgroupsv2 (all) - 7 runs, 14% failed, 100% of failures match = 14% impact pull-ci-openshift-cluster-monitoring-operator-main-e2e-hypershift-conformance (all) - 14 runs, 14% failed, 100% of failures match = 14% impact pull-ci-openshift-hypershift-main-okd-scos-e2e-aws-ovn (all) - 71 runs, 69% failed, 14% of failures match = 10% impact pull-ci-openshift-cluster-etcd-operator-main-okd-scos-e2e-aws-ovn (all) - 12 runs, 50% failed, 17% of failures match = 8% impact pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6 (all) - 62 runs, 29% failed, 6% of failures match = 2% impact pull-ci-openshift-origin-main-okd-scos-e2e-aws-ovn (all) - 38 runs, 58% failed, 5% of failures match = 3% impact pull-ci-openshift-installer-main-e2e-azure-ovn-resourcegroup (all) - 4 runs, 25% failed, 200% of failures match = 50% impact pull-ci-openshift-cloud-credential-operator-master-e2e-aws-ovn (all) - 5 runs, 80% failed, 25% of failures match = 20% impact periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 9 runs, 33% failed, 33% of failures match = 11% impact pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-local-to-shared-gateway-mode-migration (all) - 13 runs, 8% failed, 100% of failures match = 8% impact openshift-ovn-kubernetes-2790-ci-4.21-upgrade-from-stable-4.20-e2e-azure-ovn-upgrade (all) - 42 runs, 76% failed, 6% of failures match = 5% impact pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-hypershift (all) - 15 runs, 40% failed, 50% of failures match = 20% impact openshift-ovn-kubernetes-2790-ci-4.21-upgrade-from-stable-4.20-e2e-gcp-ovn-rt-upgrade (all) - 40 runs, 60% failed, 4% of failures match = 3% impact openshift-ovn-kubernetes-2790-ci-4.21-e2e-gcp-ovn-upgrade (all) - 40 runs, 73% failed, 3% of failures match = 3% impact openshift-ovn-kubernetes-2790-ci-4.21-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade (all) - 40 runs, 57% failed, 9% of failures match = 5% impact pull-ci-openshift-cluster-monitoring-operator-main-e2e-aws-ovn (all) - 18 runs, 33% failed, 17% of failures match = 6% impact periodic-ci-openshift-release-master-okd-scos-4.21-e2e-aws-ovn-upgrade (all) - 8 runs, 88% failed, 14% of failures match = 13% impact openshift-ovn-kubernetes-2790-nightly-4.21-e2e-metal-ipi-ovn-ipv6 (all) - 4 runs, 25% failed, 100% of failures match = 25% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-vsphere-ovn-upi (all) - 7 runs, 14% failed, 200% of failures match = 29% impact openshift-ovn-kubernetes-2790-periodics-e2e-aws-ovn-conformance (all) - 40 runs, 38% failed, 13% of failures match = 5% impact periodic-ci-openshift-release-master-ci-4.12-e2e-gcp-sdn-upgrade (all) - 7 runs, 29% failed, 50% of failures match = 14% impact periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-runc-upgrade (all) - 2 runs, 50% failed, 100% of failures match = 50% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-osd-ccs-gcp (all) - 8 runs, 100% failed, 38% of failures match = 38% impact periodic-ci-openshift-release-master-ci-4.20-e2e-azure-runc-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact pull-ci-openshift-cluster-version-operator-main-e2e-agnostic-ovn (all) - 10 runs, 30% failed, 33% of failures match = 10% impact pull-ci-openshift-cluster-version-operator-main-okd-scos-e2e-aws-ovn (all) - 7 runs, 57% failed, 25% of failures match = 14% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-rosa-sts-ovn (all) - 9 runs, 33% failed, 33% of failures match = 11% impact periodic-ci-openshift-release-master-okd-scos-4.21-e2e-vsphere-ovn (all) - 8 runs, 75% failed, 17% of failures match = 13% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-gcp-ovn-multi-a-a (all) - 6 runs, 67% failed, 25% of failures match = 17% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-vsphere-static-ovn (all) - 7 runs, 43% failed, 33% of failures match = 14% impact pull-ci-openshift-cluster-version-operator-main-e2e-hypershift-conformance (all) - 7 runs, 14% failed, 100% of failures match = 14% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-vsphere-ovn-multi-vcenter (all) - 2 runs, 50% failed, 100% of failures match = 50% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-upgrade-azure-ovn-multi-a-a (all) - 6 runs, 50% failed, 33% of failures match = 17% impact pull-ci-openshift-console-main-okd-scos-e2e-aws-ovn (all) - 65 runs, 46% failed, 7% of failures match = 3% impact pull-ci-openshift-installer-main-e2e-openstack-ovn (all) - 6 runs, 100% failed, 17% of failures match = 17% impact openshift-kubernetes-2484-ci-4.21-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade (all) - 60 runs, 30% failed, 17% of failures match = 5% impact pull-ci-openshift-ovn-kubernetes-release-4.20-e2e-metal-ipi-ovn-dualstack-bgp-local-gw (all) - 3 runs, 33% failed, 100% of failures match = 33% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-aws-ovn-multi-x-ax (all) - 6 runs, 17% failed, 100% of failures match = 17% impact periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-upgrade-azure-ovn-multi-a-a (all) - 6 runs, 83% failed, 20% of failures match = 17% impact openshift-kubernetes-2484-nightly-4.21-e2e-vsphere-ovn (all) - 4 runs, 25% failed, 100% of failures match = 25% impact periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn (all) - 7 runs, 14% failed, 100% of failures match = 14% impact openshift-cluster-network-operator-2806-openshift-ovn-kubernetes-2767-openshift-machine-config-operator-5317-ci-4.20-e2e-azure-ovn-upgrade (all) - 10 runs, 30% failed, 100% of failures match = 30% impact openshift-cluster-network-operator-2806-openshift-ovn-kubernetes-2767-openshift-machine-config-operator-5317-periodics-e2e-aws-ovn-conformance (all) - 10 runs, 30% failed, 167% of failures match = 50% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-aws-ovn-arm64 (all) - 8 runs, 13% failed, 200% of failures match = 25% impact periodic-ci-openshift-kni-lifecycle-agent-main-nightly-4.20-upgrade-from-stable-4.19-image-based-upgrade-e2e-parallel-conformance-baseline (all) - 1 runs, 100% failed, 100% of failures match = 100% impact periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn-upgrade-out-of-change (all) - 7 runs, 57% failed, 25% of failures match = 14% impact periodic-ci-openshift-hypershift-release-4.20-periodics-mce-e2e-agent-connected-ovn-ipv4-metal-conformance (all) - 1 runs, 100% failed, 100% of failures match = 100% impact pull-ci-openshift-ovn-kubernetes-master-e2e-azure-ovn-upgrade (all) - 12 runs, 33% failed, 25% of failures match = 8% impact pull-ci-openshift-origin-main-e2e-gcp-ovn (all) - 62 runs, 40% failed, 8% of failures match = 3% impact pull-ci-openshift-cluster-capi-operator-main-okd-scos-e2e-aws-ovn (all) - 11 runs, 18% failed, 50% of failures match = 9% impact pull-ci-openshift-kubernetes-master-e2e-aws-crun-wasm (all) - 5 runs, 60% failed, 33% of failures match = 20% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 10 runs, 30% failed, 33% of failures match = 10% impact periodic-ci-openshift-multiarch-master-nightly-4.21-upgrade-from-stable-4.20-ocp-e2e-upgrade-azure-ovn-multi-a-a (all) - 6 runs, 83% failed, 20% of failures match = 17% impact
Steps to Reproduce
1. Post a PR and have bad luck.
2. Check search.ci: https://search.dptools.openshift.org/?search=fail+%5C%5Bgithub%5C.com%2Fopenshift%2Forigin%2Ftest%2Fextended%2Fproject%2Fproject%5C.go%3A217%5C%5D&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Actual results
The test fails or flakes.
Expected results
The test passes.
Additional info
Most of the test failures that I found with search.ci are flakes. That is, the test fails at first and passes on retry.
