-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.13, 4.12, 4.14, 4.15, 4.16, 4.17, 4.18, 4.19, 4.20, 4.21
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem
CI is flaky because of test flakes and failures such as the following:
{ fail [github.com/openshift/origin/test/extended/oauth/expiration.go:173]: Unexpected error:
<*errors.StatusError | 0xc001f7c280>:
Unauthorized
{
ErrStatus:
code: 401
message: Unauthorized
metadata: {}
reason: Unauthorized
status: Failure,
}
occurred}
This particular failure comes from https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29900/pull-ci-openshift-origin-main-e2e-vsphere-ovn/1976388824435003392. Search.ci has other similar failures.
Version-Release number of selected component (if applicable)
I have seen this in 4.21, 4.20, 4.19, 4.18, 4.17, 4.16, 4.15, 4.14, 4.13, and 4.12 CI jobs.
How reproducible
Presently, search.ci shows the following stats for the past two days:
periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-upgrade (all) - 86 runs, 28% failed, 13% of failures match = 3% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-fips (all) - 7 runs, 14% failed, 100% of failures match = 14% impact pull-ci-openshift-origin-main-e2e-vsphere-ovn (all) - 61 runs, 31% failed, 16% of failures match = 5% impact periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade (all) - 78 runs, 36% failed, 7% of failures match = 3% impact openshift-kubernetes-2484-ci-4.21-e2e-azure-ovn-upgrade-out-of-change (all) - 4 runs, 75% failed, 33% of failures match = 25% impact pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6 (all) - 61 runs, 33% failed, 15% of failures match = 5% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-upgrade-fips (all) - 70 runs, 80% failed, 2% of failures match = 1% impact periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-gcp-ovn-upgrade (all) - 6 runs, 17% failed, 100% of failures match = 17% impact pull-ci-openshift-cluster-etcd-operator-main-okd-scos-e2e-aws-ovn (all) - 12 runs, 50% failed, 17% of failures match = 8% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-metal-ipi-ovn-ipv6 (all) - 7 runs, 14% failed, 100% of failures match = 14% impact periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-techpreview (all) - 7 runs, 14% failed, 100% of failures match = 14% impact pull-ci-openshift-installer-main-e2e-metal-ipi-ovn (all) - 2 runs, 50% failed, 100% of failures match = 50% impact periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-azure-aks-ovn-conformance (all) - 70 runs, 34% failed, 4% of failures match = 1% impact pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-bgp-local-gw (all) - 17 runs, 53% failed, 11% of failures match = 6% impact pull-ci-openshift-ovn-kubernetes-master-e2e-metal-ipi-ovn-dualstack-bgp (all) - 18 runs, 50% failed, 11% of failures match = 6% impact periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-upgrade-aws-ovn-arm64 (all) - 9 runs, 33% failed, 33% of failures match = 11% impact pull-ci-openshift-cluster-monitoring-operator-main-e2e-aws-ovn-techpreview (all) - 16 runs, 25% failed, 25% of failures match = 6% impact periodic-ci-openshift-release-master-ci-4.19-upgrade-from-stable-4.18-e2e-azure-ovn-upgrade (all) - 9 runs, 22% failed, 50% of failures match = 11% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-single-node (all) - 7 runs, 43% failed, 33% of failures match = 14% impact pull-ci-openshift-console-main-okd-scos-e2e-aws-ovn (all) - 65 runs, 46% failed, 3% of failures match = 2% impact periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn (all) - 7 runs, 14% failed, 100% of failures match = 14% impact periodic-ci-openshift-release-master-ci-4.16-upgrade-from-stable-4.15-e2e-aws-ovn-upgrade (all) - 9 runs, 33% failed, 33% of failures match = 11% impact periodic-ci-openshift-release-master-ci-4.21-e2e-azure-runc-upgrade (all) - 2 runs, 100% failed, 50% of failures match = 50% impact periodic-ci-openshift-release-master-okd-scos-4.21-upgrade-from-okd-scos-4.20-e2e-aws-ovn-upgrade (all) - 4 runs, 100% failed, 25% of failures match = 25% impact periodic-ci-openshift-release-master-ci-4.21-upgrade-from-stable-4.20-e2e-aws-ovn-upgrade (all) - 87 runs, 39% failed, 3% of failures match = 1% impact openshift-kubernetes-2484-nightly-4.21-e2e-aws-ovn-upgrade-fips (all) - 42 runs, 36% failed, 20% of failures match = 7% impact periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-gcp-ovn-rt-upgrade (all) - 8 runs, 63% failed, 20% of failures match = 13% impact periodic-ci-openshift-release-master-ci-4.21-e2e-azure-ovn-upgrade (all) - 80 runs, 46% failed, 5% of failures match = 3% impact periodic-ci-openshift-multiarch-master-nightly-4.20-ocp-e2e-aws-ovn-techpreview-multi-a-a (all) - 6 runs, 33% failed, 50% of failures match = 17% impact periodic-ci-openshift-release-master-ci-4.17-upgrade-from-stable-4.16-e2e-aws-ovn-upgrade (all) - 7 runs, 14% failed, 100% of failures match = 14% impact openshift-ovn-kubernetes-2790-ci-4.21-e2e-azure-ovn-upgrade (all) - 42 runs, 90% failed, 3% of failures match = 2% impact pull-ci-openshift-ovn-kubernetes-master-e2e-aws-ovn-upgrade-local-gateway (all) - 13 runs, 15% failed, 100% of failures match = 15% impact openshift-ovn-kubernetes-2790-nightly-4.21-e2e-aws-ovn-upgrade-fips (all) - 40 runs, 68% failed, 4% of failures match = 3% impact periodic-ci-openshift-multiarch-master-nightly-4.21-ocp-e2e-upgrade-aws-ovn-multi-a-a (all) - 6 runs, 33% failed, 50% of failures match = 17% impact periodic-ci-openshift-release-master-ci-4.21-e2e-gcp-ovn-upgrade (all) - 85 runs, 33% failed, 7% of failures match = 2% impact periodic-ci-openshift-release-master-nightly-4.21-e2e-aws-ovn-proxy (all) - 11 runs, 18% failed, 50% of failures match = 9% impact pull-ci-openshift-ovn-kubernetes-release-4.18-4.18-upgrade-from-stable-4.17-e2e-aws-ovn-upgrade (all) - 9 runs, 67% failed, 17% of failures match = 11% impact periodic-ci-openshift-release-master-ci-4.21-e2e-azure-ovn-upgrade-out-of-change (all) - 8 runs, 63% failed, 20% of failures match = 13% impact periodic-ci-openshift-release-master-nightly-4.19-e2e-metal-ipi-ovn-ipv6 (all) - 8 runs, 13% failed, 100% of failures match = 13% impact pull-ci-openshift-origin-main-e2e-aws-ovn-fips (all) - 60 runs, 33% failed, 5% of failures match = 2% impact periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-upgrade-fips (all) - 60 runs, 27% failed, 6% of failures match = 2% impact periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance (all) - 72 runs, 14% failed, 20% of failures match = 3% impact periodic-ci-openshift-cloud-credential-operator-release-4.22-periodics-e2e-azure-manual-oidc (all) - 2 runs, 100% failed, 50% of failures match = 50% impact
Steps to Reproduce
1. Post a PR and have bad luck.
2. Check search.ci: https://search.dptools.openshift.org/?search=fail+%5C%5Bgithub%5C.com%2Fopenshift%2Forigin%2Ftest%2Fextended%2Foauth%2Fexpiration%5C.go%3A173%5C%5D%3A+Unexpected+error%3A&maxAge=48h&context=1&type=bug%2Bissue%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Actual results
The test fails or flakes.
Expected results
The test passes.
Additional info
In the search.ci results, most of the test failures are flakes. That is, the test usually fails once, then passes on retry.