Details
-
Bug
-
Resolution: Done
-
Normal
-
None
-
None
-
None
-
False
-
OCPSTRAT-140 - [Docs] Onboarding New Providers/Platforms (Phase 3)
Description
The report command is returning 'pass' after applying filters - mainly for OPCT failues (custom)
> Processed Summary < Total tests by conformance suites: - kubernetes/conformance: 358 - openshift/conformance: 3453 Result Summary by conformance plugins: - 10-openshift-kube-conformance: - Status: failed - Total: 1 - Passed: 0 - Failed: 1 - Timeout: 0 - Skipped: 0 - Failed (without filters) : 1 - Failed (Filter SuiteOnly): 0 - Failed (Filter CI Flakes): 0 - Status After Filters : pass - 20-openshift-conformance-validated: - Status: failed - Total: 1 - Passed: 0 - Failed: 1 - Timeout: 0 - Skipped: 0 - Failed (without filters) : 1 - Failed (Filter SuiteOnly): 0 - Failed (Filter CI Flakes): 0 - Status After Filters : pass
It used happen when custom failures, created by OPCT plugin, on the runtime. The filters will exclude those failures while it's match for the filter evaluation.
The filter engine must be patched to accept those kind of errors, mainly created by OPCT runtime.
We expected to fail, when 'runtime' failures are detected.
=> 10-openshift-kube-conformance: (1 failures, 0 failures filtered, 0 flakes) --> [verbose] Failed tests detected on archive (without filters): [opct] [runner] default error handler: openshift-tests did not created JUnit file(s) --> [verbose] Failed tests detected on suite (Filter SuiteOnly): <empty> --> Failed tests to Review (without flakes) - Immediate action: <empty> --> Failed flake tests - Statistic from OpenShift CI <empty> => 20-openshift-conformance-validated: (1 failures, 0 failures filtered, 0 flakes) --> [verbose] Failed tests detected on archive (without filters): [opct] [runner] default error handler: openshift-tests did not created JUnit file(s)
