Goal

Each SIG group got the Goal to CI pass rate to 99%+, to help on this goal, we plan to build an automated workflow to identify, quarantine, and manage failed test cases immediately after test execution, preventing them from impacting subsequent test cycles and enabling faster issue triaging.

User Stories

• If it has a failure rate higher than 20% in the last 3 days, as a SIG member, I need failed test cases to be quarantined automatically.
• If it has a failure rate higher than 5% in the last 2 weeks, as a SIG member, I need failed test cases to be quarantined automatically. 
• For quarantine the failed test cases automatically, as a SIG member, I need alerts for quarantined test cases: auto-notifications via email and slack to relevant teams with alert includes failures details(job, test environment etc.)
• For quarantine the failed test cases automatically, the PR should add the label: auto-quarantined that can help us to filter the PRs. 
• In SIG quarantine weekly meeting, sync with team for automate quarantine workflow and update the upstream document with this change: https://github.com/kubevirt/kubevirt/blob/main/docs/quarantine.md

Non-Requirements

• List of things not included in this epic, to alleviate any doubt raised during the grooming process.

Notes

• If the failure rate continues to fall short of requirements(each SIG: 99.15% pass rate), we may consider to quarantine each failing case if they are failed twice a day or two days we see the same failures until the target is met. Additionally, should there be big concerns regarding test regression or code coverage, then please prioritize resolving failed cases promptly to ensure each SIG group achieves its objectives.