Record the time it takes to invoke a sync RPC, to the following details

• Send a request down until it is serialized
• Send the req to the receiver
• De-serialization of the request until it is received by RequestCorrelator
• Time to dispatch the request into the application
• Sending the response down the stack, until it is serialized
• Time to send the response to the sender
• De-serialization of the response and unblocking the sender

The goal of this is to see where the total RTT time is spent and to reduce the times. It is also beneficial to see the impact of changing parameters (e.g. invoker count, request size etc).

The way to do this is to create a new header (RttHeader?) that's attached to each message (also each message in a batch).The header contains a number of timestamps that are set by the various components, e.g. before serialization, when dispatching into the application etc.

A ProbeHandler should expose the stats (e.g. min-max-avg-mean times).

Investigate whether Byteman should be used to set certain timestamps in the header; the advantage would be that no code changes are necessary.

This only works for members residing on the same host; when different hosts are used, wall-clock times need to be tightly synchronized (not recommended).