Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-2173

Eliminate E2E Retry Loophole

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • Eliminate E2E Flake Loophole
    • Product / Portfolio Work
    • 33% To Do, 0% In Progress, 67% Done
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Selected
    • None
    • None
    • None

      OpenShift e2e testing has allowed failing tests to retry for years. Doing this is dangerous as it allows a test to start flaking significantly more than previous releases without anyone being able to see. We need to start driving towards eventual elimination of this mechanism by improving our insight into flaking regressions, disabling the mechanism for new tests, and driving down the backlog of tests that suffer these flakes.

      DISCLAIMER: this does not imply removal of the ability to retry a test in the same job run. There are separate initiatives around this to help improve signal with less cost by attempting a test multiple times. This epic is purely around NOT hiding the failures with a successful retry. We want to either fail and not retry, or treat retries as disparate test results without hiding the failures.

              rhn-engineering-dgoodwin Devan Goodwin
              rhn-engineering-dgoodwin Devan Goodwin
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: