TCP (in TcpConnection) sends the length of the data first, then the data (the serialized message, or message list).

It turns out that DataOutputStream.writeInt() is slow when used over a SocketOutputStream: every byte of the int is sent separately, resulting in a temp byte array of length=1 being created and sent in SocketOutputStream 4 times!

In temp perf measurements, sending the int took up to 70% of the entire send (including the data)!

This happens in JDK 11. The code in more recent JDKs (19, for instance) uses a writeBuffer (of length 8), into which they write the bytes making up an int (or long), and then send the writeBuffer, so there's no issue in more recent JDKs. However, we're stuck with 11 for quite a while, so let's fix this now.

Solutions:

• Use a BufferedOutputStream over the SocketOutputStream. This is enabled when TCP.buffered_output_stream_size is > 0. The downside of this is that messages are not sent right away when multiple threads are sending over the same TcpConnection, increasing latency. The message batch is sent when either the BufferedOutputStream is full, or there's no more sender thread (last sender thread sends).
• Create a temp array of data.length plus 4, into which we write the length, followed by the data, then write the array (only 1 write instead of 2). Measurements showed that this was much faster. However, we have the temp array creation, which increases memory pressure.
• Create a byte array of 4 length in TcpConnection, into which the length of the data is written, then write this first array to the output stream, the the data array. This is also much faster, but has 2 writes, which is also a bit inefficient
• Omit sending the length. Investigate whether this is feasible, also for TCP_NIO2.