Ideas for how to improve the throughput of the Journal Store

We had some discussions with the Artemis team (fnigro ) around optimising the performance of our Journal Store which resulted in some ideas about how to improve throughput:

1. Consuming less CPU with a larger buffer timeout (and other tunable journal parameters) using linux tools such as fio and strace. We also discussed strategies for avoiding "write amplification".
2. On a fast disk, the calling appender thread can avoid sleeping while waiting for the SSD write and flush operations to finish by spin waiting while the append operation executes ("you snooze, you loose"), i.e. consuming CPU in the fast disk case can significantly out perform having to sleep while the output operation proceeds.
3. Internally Artemis uses an appending executor [1] based on a SynchronousQueue which could be implemented more efficiently using a better thread pool [2], in particular under load it can create a huge number of threads. The narayana use case really only requires a fixed pool of three threads: the appender (serializes append requests and sends them through the timed buffer thread), the compactor and the new file creator/intialiser (there are others such as the timed buffer and libaio poller threads), and we can analyse these using flame graphs and stackdumps, so the experiment here is to change Artemis to use different thread pools optimised for the transaction use case and, if useful, allow it to be configurable.

[1] https://github.com/apache/activemq-artemis/blob/main/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalImpl.java#L308
[2] https://github.com/apache/activemq-artemis/blob/main/artemis-journal/src/main/java/org/apache/activemq/artemis/core/journal/impl/JournalImpl.java#L2964