We need to be able to tag individual messages with a quality-of-service bit: ASYNC. This means that, when such a message is received, it can be delivered asynchronously, e.g. by a thread from a separate threadpool (compared this to http://jira.jboss.com/jira/browse/JGRP-181).
Examples for OOB messages:

• ACKs (unicast)
• XMIT requests (NAKACK)
• Credit replenishment messages (FC)
• Failure detection: heartbeats and are-you-alive messages, plus acks

We might even introduce priority based message delivery

One example for FC:

• We have members A and B
• A and B continuously invoke put()s on the TreeCache in synchronous mode
• When B receives a put() request, it applies it and wants to send the response. However, assume that the response is blocked in FC.down() because we don't have enough credits available to send the response to A
• Now A sent a REPLENISH message to B, but B is still stuck in the FC.down() method, which blocks the up thread !
• Therefore B cannot handle the replenishment message from A and therefore won't unblock: deadlock !
• If we could deliver the REPLENISH message from A to B on a separate thread, B would receive the REPLENISH message and unblock the FC.down() method