Uploaded image for project: 'OCP Technical Release Team'
  1. OCP Technical Release Team
  2. TRT-1825

Try to detect new improper tests during RA PR-commenting

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Major Major
    • None
    • None
    • None
    • None
    • None
    • None

      We have an issue where new tests often don't reliably produce success results, or have other issues like dynamic components in the test name (e.g. a namespace).

      Could risk analysis be configured to report on all never-before-seen tests in presubmits? And maybe a link/reminder of our test standards (no dynamic names, must always produce a result, renames should be accounted for)

      Benefits of just commenting the new test names:

      1. Developer might notice dynamic names in the list
      2. We can search github for the whole test name and find the PR easily that added it (or push it to bigquery?)

      This would potentially let developers see the tests they're creating. We might also be able to apply heuristics to know that things aren't working correctly? 

      e.g. we only saw a test name once across all presubmits, or we saw failures but no successes, etc.

      Outcomes:

      • Risk analysis comment would include new tests to increase visability and searchability 
      • Risk analysis would mark the risk High if a new test failed as we don't want new failing tests to enter the system without validation

              lmeyer@redhat.com Luke Meyer
              stbenjam Stephen Benjamin
              None
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: