Sipke to check the effort to pipe the failed tests into a new serial execution to try to decrease or make the second validation of failures from the original suites. The result can be used as a new filter ( SPLAT-831 ) to improve the quality of the results provided by the CLI.

We are talking with TRT to understand how they used to validate the failed jobs in CI, and how we can use those ideas on OPCT to provide better feedback to the partner in an automated fashion - before consuming resources com Red Hat (Support and Engineering time, CI Infra, etc). One of the ideas that were brought up was that we need to collect more data to take more accurate decisions with regard the failures, so, for example, may need to ask the partner to run the certification environment almost 10x in new clusters and then analyze the data looking for flakes and failures. The other idea was to submit the metadata files to our CI environment to be consumed and available somehow by Sippy.

We are evaluating all those ideas, according to the goals of the Certification Program. Some of the principles and goals are:

• Decrease the overhead for the long-tail partners trying to certify their environment
• Reuse existing tools
• Empower the partner with information to be able to identify the issues from their end before sending them to us
• Automate the most possible