-
Feature Request
-
Resolution: Done
-
Major
-
6.2.0.Final
-
None
-
NEW
-
NEW
Watch the presentation of Josh Block called Performance Anxiety on parleys: https://www.parleys.com/tutorial/performance-anxiety
Running a single benchmark just once is unreliable:
- 2 JVM processes on the same hardware for the same code can behave very differently performance wise (= score calculation count per second)
- If the randomSeed isn't fixed (for example in PRODUCTION mode), a different randomSeed will influence score quality to certain degree.
Our benchmarks needs to be to show the impact of this, by easily allowing to do every single benchmark n times. The benchmark report should show the average, the minimum & maximum (maybe even with a candle stick diagram?) and maybe even the raw result of every separate single benchmark run. Requirements needs to be discussed further before implementation starts.
After this is implemented, this feature can be used to validate or invalidate the information in old blog posts that just did 1 single benchmark and presumed it was representative:
http://www.optaplanner.org/blog/tags/production/
- causes
-
PLANNER-473 Benchmark report: Best score problem statistic with subSingleCount mixes median and average score
-
- Resolved
-
- is related to
-
-
- Open
-
-
PLANNER-438 Migrate benchmarker report to newest version of Bootstrap v4
-
- Open
-
-
PLANNER-443 Add a reliability tab to summary statistics in benchmarker report
-
- Resolved
-
-
-
- Resolved
-
-
-
- Open
-
-
-
- Open
-
-
-
- Open
-
