Benchmarks are usually long-running processes. During the runtime of a benchmark, lots of things can happen - you may lose power on your laptop, you may accidentally kill the benchmark process... Those have already happened to me and when they do, I lose the hours of benchmarks that have run so far but haven't yet generated a HTML report.

What I propose is the following:

• After each benchmark in a suite finishes, some kind of intermediate file is written, describing exactly which other benchmarks are left to do. It may also need to store the results of the already finished benchmarks, so that in the future we can create the charts etc.
• A new API is added to Planner that allows to resume the benchmark from the above intermediate file.
• Possibly also another API for "suspending" the benchmarking process, ie. persisting it into the intermediate file and halting.