The following scenario/example affects performance:

• A batch of 1000 requests is received and dispatched to the application (each message is a synchronous RPC)
• Processing the requests in the batch is done one-by-one, and a response is sent for each request.

This means that the thread which delivered the request needs to send the response before it can unwind and process the next request in the batch.

If the dispatching of a request took 1ms and the sending of the response also 1ms, then processing a single request would take 2ms. Processing the entire batch would therefore take 2000ms (assuming that the application doesn't use asynchronous processing).

In addition, we have other synchronous message sends that might occur on the same thread:

• UNICAST3 might send an ACK for a unicast message
• Flow control (UFC/MFC) might send credits to the sender of a message or message batch
• NAKACK2/UNICAST3 might resend a message as a result of receiving a retransmission request
• UNICAST3: send request for first message (SEND_FIRST_SEQNO)

All of these add to the processing time of a message.

The goals of this issue are:

1. Reduce the time required to handle incoming messages (main goal) by sending messages asynchronously
2. Reduce the number of messages sent (minor goal) by piggybacking

Sending messages asynchronously

If we sent the response to a request asynchronously (on a separate thread), then in the processing of the scenario above, the time would be reduced to 1000ms: the delivering thread would not have to send the response, but unwind immediately. Note that the sending cost could possibly involve blocking on credits in flow control, on the send queue in the bundler etc, so the send-cost might even be higher.

There should be a downAsync(Message) in Protocol, which returns immediately. The implementation could use a send-queue and a send thread which continually removes elements from the send-queue and sends them, possibly attaching piggybacked messages to each message (as many as possible, but only 1 for each protocol)

Piggybacking

This is the idea that - when sending a message M to target P - on the receive path, we simply add M to a queue. When an actual message M' to P is sent, we piggyback (attach) M to M' if possible. It is of course not possible to attach M to M' if both messages have a payload, but if M' has a payload and M simply a header, this would work. Alternatively, we could tun M' into a BatchMessage, including M' and M. When there is no actual message M' sent, the async send thread (above) could simply send a queued message.

This entire behavior should be configurable; we should be able to turn it on or off even at run time (e.g. via probe.sh).