Uploaded image for project: 'OptaPlanner'
  1. OptaPlanner
  2. PLANNER-401

Statistical benchmarking: run each single benchmark n times and average the results

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Done
    • Major
    • 6.4.0.Beta1
    • 6.2.0.Final
    • optaplanner-benchmark
    • None
    • NEW
    • NEW

    Description

      Watch the presentation of Josh Block called Performance Anxiety on parleys: https://www.parleys.com/tutorial/performance-anxiety

      Running a single benchmark just once is unreliable:

      • 2 JVM processes on the same hardware for the same code can behave very differently performance wise (= score calculation count per second)
      • If the randomSeed isn't fixed (for example in PRODUCTION mode), a different randomSeed will influence score quality to certain degree.

      Our benchmarks needs to be to show the impact of this, by easily allowing to do every single benchmark n times. The benchmark report should show the average, the minimum & maximum (maybe even with a candle stick diagram?) and maybe even the raw result of every separate single benchmark run. Requirements needs to be discussed further before implementation starts.

      After this is implemented, this feature can be used to validate or invalidate the information in old blog posts that just did 1 single benchmark and presumed it was representative:
      http://www.optaplanner.org/blog/tags/production/

      Attachments

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. PLANNER-473 Benchmark report: Best score problem statistic with subSingleCount mixes median and average score

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is related to

          Feature Request - Feature requests from customers and/or users PLANNER-460 Add sub single benchmark data point distribution density to single statistic charts

          • Optional - The request should be considered desirable but is not an immediate necessity.
          • Open

          Task - Represents a small unit of work that is not end-user facing. PLANNER-438 Migrate benchmarker report to newest version of Bootstrap v4

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Open

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) PLANNER-443 Add a reliability tab to summary statistics in benchmarker report

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) PLANNER-475 Reproducible statistical benchmarking

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Resolved

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) PLANNER-446 Detect sub single benchmark outliers based on ACC and report them

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Open

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) PLANNER-428 Run SubSingleBenchmarks in separate JVMs

          • Optional - The request should be considered desirable but is not an immediate necessity.
          • Open

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) PLANNER-429 Migrate box plots in benchmarker to violin plots once JFreeChart supports them

          • Optional - The request should be considered desirable but is not an immediate necessity.
          • Open

          Activity

            People

              oskopek Ondrej Skopek (Inactive)
              gdesmet@redhat.com Geoffrey De Smet (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: