OpenShift e2e testing has allowed failing tests to retry for years. Doing this is dangerous as it allows a test to start flaking significantly more than previous releases without anyone being able to see. We need to start driving towards eventual elimination of this mechanism by improving our insight into flaking regressions, disabling the mechanism for new tests, and driving down the backlog of tests that suffer these flakes.

DISCLAIMER: this does not imply removal of the ability to retry a test in the same job run. There are separate initiatives around this to help improve signal with less cost by attempting a test multiple times. This epic is purely around NOT hiding the failures with a successful retry. We want to either fail and not retry, or treat retries as disparate test results without hiding the failures.