Uploaded image for project: 'OpenShift Virtualization'
  1. OpenShift Virtualization
  2. CNV-71660

tier2 test suite reports incorrect "Tests Run" count in summary-log.txt (includes skipped tests)

XMLWordPrintable

    • Product / Portfolio Work
    • 0.42
    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • None

      Description of problem:

        The tier2 test suite reports inconsistent test counts between the JUnit XML file and the summary-log.txt. The JUnit file correctly shows 174 tests run (175 total - 1 skipped), but the summary-log.txt incorrectly reports 175 tests
        run.
      
        The summary-log.txt generation logic appears to be using the 'tests' attribute directly from the JUnit XML header for tier2 suite instead of calculating 'tests - skipped'. This is inconsistent with how other test suites
        (compute/network/storage) work, where the 'tests' attribute already excludes skipped tests.
        

      Version-Release number of selected component (if applicable):

        ocp-virt-validation-checkup v4.19
        

      How reproducible:

        Always - occurs whenever tier2 test suite runs with at least one skipped test
        

      Steps to Reproduce:

        1. Run ocp-virt-validation-checkup with tier2 test suite enabled
        2. Ensure at least one test is skipped during execution
        3. Retrieve results from the PVC after job completion
        4. Compare tier2/junit.results.xml with summary-log.txt
        

      Actual results:

        JUnit XML (tier2/junit.results.xml):
          <testsuite name="pytest" errors="0" failures="0" skipped="1" tests="175" ...>
      
          Calculation: 175 total tests - 1 skipped = 174 tests run
      
        summary-log.txt:
          =================
          Summary for tier2
          =================
          Tests Run: 175          <-- INCORRECT (includes skipped test)
          Tests Passed: 175
          Tests Failed: 0
          Tests Skipped: 1
      
        The "Tests Run" count incorrectly includes the skipped test (175 instead of 174).
        

      Expected results:

        summary-log.txt should report:
          =================
          Summary for tier2
          =================
          Tests Run: 174          <-- CORRECT (excludes skipped test)
          Tests Passed: 174
          Tests Failed: 0
          Tests Skipped: 1
      
        For pytest-based test suites (tier2), the Tests Run count should be calculated as:
          Tests Run = tests attribute - skipped attribute
          Tests Run = 175 - 1 = 174
        

      Additional info:

        Root cause: The summary generation logic needs to handle different JUnit XML formats:
      
        1. Ginkgo format (compute/network/storage/ssp):
           - The 'tests' attribute already contains only executed tests
           - Skipped tests are not included in the count
      
        2. pytest format (tier2):
           - The 'tests' attribute contains ALL tests (executed + skipped)
           - Must subtract 'skipped' attribute to get actual tests run
      
        Impact:
        - ConfigMap shows misleading test run counts for tier2 suite
        - Inconsistency between JUnit files and summary makes result validation difficult
      
      Logs attached.
        

              ocohen@redhat.com Oren Cohen
              rlobillo Ramón Lobillo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: