Goal:

• Research for existing tooling to provide zero false-positive test failures in OPCT report in openshift conformance plugin results (20-openshift-conformance-validated)

Description:

Currently, OPCT apply filters[1] in the reporting pipeline to provide more clear failures in the provider's execution, although it still reports some issues that would not be related to the environment, as you can see in this example[2].

This card aims to provide information to reviewers (partners, cert-ops, or engineering) of existing failures, targeting to ignore non-related issues.

Some sources can be:

• Sippy risk-analysis
• insights rules for OCP
• any other?

Example of OPCT report that could be used as a reference "zero failures" target: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_release/40390/rehearse-40390-periodic-ci-redhat-openshift-ecosystem-provider-certification-tool-main-4.13-platform-none-vsphere/1680792017874980864/artifacts/platform-none-vsphere/opct-test-results/artifacts/opct-report.html 

Engineering references:

[1] opct filters

• suite only
• baseline
• flake check

https://redhat-openshift-ecosystem.github.io/provider-certification-tool/dev/#cli-result-filters 

[2] example of using baseline

• Without baseline

  $ ./opct-dev report 20230721/202307201345_sonobuoy_de7570b1-9fe6-4e55-a821-e1bf3549d7bd.tar.gz 
  
   Result Summary by conformance plugins:
  
   - 20-openshift-conformance-validated:
     - Status: failed
     - Total: 3923
     - Passed: 1664 (42.42%)
     - Failed: 80 (2.04%)
     - Timeout: 0 (0.00%)
     - Skipped: 2179 (55.54%)
     - Failed (without filters) : 80 (2.04%)
     - Failed (Filter SuiteOnly): 46 (1.17%)
     - Failed (Filter CI Flakes): 39 (0.99%)
  

• With baseline filter (–diff) - 13 failed tests have been removed from the 'baseline'.

  ./opct-dev report 20230721/202307201345_sonobuoy_de7570b1-9fe6-4e55-a821-e1bf3549d7bd.tar.gz --diff 4.13.4-20230717-HighlyAvailable-vsphere-None.tar.gz
   
  Result Summary by conformance plugins:
   
   - 20-openshift-conformance-validated:
     - Status: failed
     - Total: 3923
     - Passed: 1664 (42.42%)
     - Failed: 80 (2.04%)
     - Timeout: 0 (0.00%)
     - Skipped: 2179 (55.54%)
     - Failed (without filters) : 80 (2.04%)
     - Failed (Filter SuiteOnly): 46 (1.17%)
     - Failed (Filter Baseline) : 27 (0.69%)
     - Failed (Filter CI Flakes): 26 (0.66%)