Setting up a benchmarker currently takes an hour or so. Many users are too time pressured for that (they're too busy tracking down their performance issue... sigh). So we should make it quicker for them to set up benchmarker.

Something like this should work:

SolverFactory sf = SolverFactory.createFromXmlResource(".../nqueensSolverConfig.xml");
CloudBalance problem = ...;
PlannerBenchmarkFactory bf = PlannerBenchmarkFactory.createFromSolverFactory(sf, problem);
bf.buildPlannerBenchmark();

If anyone has a perf problem and then still doesn't use the Benchmarker, well then ...

Details:

• default to 30 secs warmup
• default to "local/benchmarkReport" for benchmark report output. (One can argue that this should default to "target/benchmarkReport" but as that's cleaned during "mvn clean", I wouldn't mind promoting the local directory in .gitignore approach (which is great).)
• PLANNER-724 Definitely including the BEST_SCORE problem statistic. We can worry later if we want to add the others too (due to perf impact).
• Yes, this benchmark report has only 1 solver configuration and 1 dataset, but it does show the best score graph, the perf, the warnings, etc