Unless I'm just doing something wrong, benchmarking without a logger causes the exception shown below.

I've looked at the code, and the logging level seems to be used in exactly two places. One is to show it in the benchmark result. The other is when building the list of warnings in BenchmarkReport.fillWarningList(). Since that method checks whether the logging level is null, that makes me think it's not supposed to be required, making this more likely to be a bug than a feature. (Incidentally, that null check can be refactored away.)

Exception in thread "main" java.lang.IllegalStateException: Logging level for loggerName (org.optaplanner.core) cannot be determined.
        at org.optaplanner.benchmark.impl.result.PlannerBenchmarkResult.resolveLoggingLevel(PlannerBenchmarkResult.java:315)
        at org.optaplanner.benchmark.impl.result.PlannerBenchmarkResult.initSystemProperties(PlannerBenchmarkResult.java:288)
        at org.optaplanner.benchmark.impl.DefaultPlannerBenchmark.benchmarkingStarted(DefaultPlannerBenchmark.java:116)
        at org.optaplanner.benchmark.impl.DefaultPlannerBenchmark.benchmark(DefaultPlannerBenchmark.java:97)
        at org.optaplanner.benchmark.impl.DefaultPlannerBenchmark.benchmarkAndShowReportInBrowser(DefaultPlannerBenchmark.java:424)