Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-43178

Virt: non-failing T1 lanes

XMLWordPrintable

    • virt-no-retries
    • 77
    • Hide
      • For the 4.18 release, the failure rate (the number of times a PR fails Tier 1 tests before it is merged) must be less than 5%
      • This is the number of times a CI run failed divided by the total number of CI executions
      • Only the last code change to a PR should be evaluated. The assumption is that if the code was changed there were bugs that were being addressed.
      • Gating Tests should always PASS.
      Show
      For the 4.18 release, the failure rate (the number of times a PR fails Tier 1 tests before it is merged) must be less than 5% This is the number of times a CI run failed divided by the total number of CI executions Only the last code change to a PR should be evaluated. The assumption is that if the code was changed there were bugs that were being addressed. Gating Tests should always PASS.
    • Green
    • To Do
    • CNV-38613 - 2024 Code Quality
    • CNV-386132024 Code Quality
    • 50% To Do, 38% In Progress, 13% Done
    • dev-ready, doc-ready, po-ready, qe-ready, ux-ready
    • Hide

      dev is working on this....

      Show
      dev is working on this....

      Goal

      • sig-compute should always pass when testing good merged code on kubevirtci.
        • Identify and Prioritize High Priority Virt T1 Tests for Downstream Builds.
          • Use the "Conformance" label to select the Virt T1 Tests for Gating.
        • Remove and/or Rewrite the tests after reviewing the tests.
      • Ensure the Non Productized Tests from Tier1 Jobs are skipped.
      • Create Full T1 Scheduled job runs for kubevirt-compute-ocs.
      • Update existing jobs to pick the tests labeled with Conformance.

       
      Discussion Doc:

      • As a KubeVirt developer, I want to trust my tests. When they fail, I need to know that I introduced a bug. I must not assume that "this may be a flake".
        • As of 2024-06-19, out of 31 PRs that were merged in the recent week, only 13 merged cleanly. The rest (18) required a /retry to pass. As a KubeVirt developer I would accept a failure rate of 5%, but no more.
      • As an OpenShift Virtualization developer, I would like to see my tests passing in downstream CI without any hiccup.
      • As an OpenShift Virtualization admin, I would like to receive weekly updates of my operator. I would like it to pass continuous integration smoothly and without human intervention - no failure analysis and no manual wavers.

      Non-Requirements

      • List of things not included in this epic, to alleviate any doubt raised during the grooming process.

      Notes

      • Any additional details or decisions made/needed

          1.
          upstream roadmap issue Sub-task New Normal Unassigned
          2.
          upstream design Sub-task New Normal Unassigned
          3.
          upstream documentation Sub-task New Normal Unassigned
          4.
          upgrade consideration Sub-task New Normal Unassigned
          5.
          CEE/PX summary presentation Sub-task New Normal Unassigned
          6.
          test plans in polarion Sub-task New Normal Unassigned
          7.
          automated tests Sub-task New Normal Unassigned
          8.
          downstream documentation merged Sub-task New Normal Unassigned

              sgott@redhat.com Stuart Gott
              dkenigsb@redhat.com Dan Kenigsberg
              Kedar Bidarkar Kedar Bidarkar
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: