-
Bug
-
Resolution: Unresolved
-
Minor
-
CNV v4.19.z, CNV v4.20.z
-
None
-
Product / Portfolio Work
-
0.42
-
False
-
-
False
-
None
-
-
None
Description of problem:
The tier2 test suite reports inconsistent test counts between the JUnit XML file and the summary-log.txt. The JUnit file correctly shows 174 tests run (175 total - 1 skipped), but the summary-log.txt incorrectly reports 175 tests run. The summary-log.txt generation logic appears to be using the 'tests' attribute directly from the JUnit XML header for tier2 suite instead of calculating 'tests - skipped'. This is inconsistent with how other test suites (compute/network/storage) work, where the 'tests' attribute already excludes skipped tests.
Version-Release number of selected component (if applicable):
ocp-virt-validation-checkup v4.19
How reproducible:
Always - occurs whenever tier2 test suite runs with at least one skipped test
Steps to Reproduce:
1. Run ocp-virt-validation-checkup with tier2 test suite enabled 2. Ensure at least one test is skipped during execution 3. Retrieve results from the PVC after job completion 4. Compare tier2/junit.results.xml with summary-log.txt
Actual results:
JUnit XML (tier2/junit.results.xml):
<testsuite name="pytest" errors="0" failures="0" skipped="1" tests="175" ...>
Calculation: 175 total tests - 1 skipped = 174 tests run
summary-log.txt:
=================
Summary for tier2
=================
Tests Run: 175 <-- INCORRECT (includes skipped test)
Tests Passed: 175
Tests Failed: 0
Tests Skipped: 1
The "Tests Run" count incorrectly includes the skipped test (175 instead of 174).
Expected results:
summary-log.txt should report:
=================
Summary for tier2
=================
Tests Run: 174 <-- CORRECT (excludes skipped test)
Tests Passed: 174
Tests Failed: 0
Tests Skipped: 1
For pytest-based test suites (tier2), the Tests Run count should be calculated as:
Tests Run = tests attribute - skipped attribute
Tests Run = 175 - 1 = 174
Additional info:
Root cause: The summary generation logic needs to handle different JUnit XML formats:
1. Ginkgo format (compute/network/storage/ssp):
- The 'tests' attribute already contains only executed tests
- Skipped tests are not included in the count
2. pytest format (tier2):
- The 'tests' attribute contains ALL tests (executed + skipped)
- Must subtract 'skipped' attribute to get actual tests run
Impact:
- ConfigMap shows misleading test run counts for tier2 suite
- Inconsistency between JUnit files and summary makes result validation difficult
Logs attached.
- is related to
-
CNV-68306 OCP Virtualization self-certification package for Cloud providers (GA) - UI and disconnected
-
- New
-