Uploaded image for project: 'JBoss Transaction Manager'
  1. JBoss Transaction Manager
  2. JBTM-2419

Reduce the number of performance regression tests that run against a PR

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Minor
    • 5.11.0.Final
    • 5.1.1
    • Testing
    • None

    Description

      We run our JMH tests multiple times in order to estimate run-to-run variance. The JMH docs state:

      JVMs are complex systems, and the non-determinism is inherent for them. This requires us to always account the run-to-run variance as the one of the effects in our experiments. Forking aggregates the results across several JVM launches.

      Currently we set the number of forks to 10. We run each test 6 times with each run lasting 1 min. Since we have 10 tests this means it takes 10 * 10 * 6 minutes for the benchmarks to complete. In the future we want to keep adding new performance tests and the time taken to run a PR will become prohibitive. I would like to propose one of the following solutions:

      • reduce the number of forks;
      • run a subset of performance tests on each PR and then run them all once a week or so on the main build

      My preference is to reduce the number of forks to 3. Note that the PR requester has the option to disable performance tests for his PR (by including !PERF in the comment section).

      Attachments

        Activity

          People

            mkunwar1 Mayank Kunwar (Inactive)
            rhn-engineering-mmusgrov Michael Musgrove
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: