qa tests should support in-process recovery

The qa (formerly dtf) testsuite spawns multiple client/server processes for each test. As currently configured, these share an objectstore dir and the recovery manager runs as a separate process. This makes them unsuitable for testing objectstore implementations that have in-process state and expect process exclusive access to their on-disk state i.e. only support in-process recovery manager. Now that we actually plan to create such implementations that's become a problem.

The testsuite is unwieldy enough as it is, so we don't want to clone and modify tests, but rather have existing tests run with a dedicated objectstore and recovery manager per process rather than per test. This should be configurable, such that the tests may be run in either external or in-process recovery configuration.

—

Modify the test harness to provide a uniq objectstore dir location to each spawned process. Allow crash recovery tests to selectively override this, as pre and post crash processes must use the same store or there won't be any state to recover. This requires tweaking the crashrec tests themselves to supply a correspondence identifier to the test harness so it can match pre and post crash processes. Spawned processes likewise need uniq/matching port values to avoid listener port conflicts between concurrent processes and ensure the recovery IORs are valid.

Provide an optionally activated wrapper that will suppress the standalone recovery process and start/stop an in-process recovery system for each spawned task in the test. Use the directory and port values supplied to configure this. When the wrapper is not present, default behaviour of shared dir with external rec mgr will apply.