New protocol to prevent threads from the thread pool (handling incoming messages) to block on a send on the same thread.

Example:

• Thread T from the thread pool is assigned to send message M up the stack
• Flow control passes M up to the application
• In receive(), the application sends message M-rsp. This could for example be a response to the sender, a multicast to the group etc
• When receive() returns, flow control might also send a credit response M-cred to the sender

All of this happens before a new message can be processed. If either M-rsp or M-cred blocks on sending (TCP), then T is blocked, too, and cannot be used for the processing of another message. This can lead to thread pool exhaustion.

Solution

We can detect that an incoming thread T is used to send M-rsp and M-cred and tag these 2 as DONT_BLOCK (JGRP-2771). This prevents these messages from blocking on a full queue (they're discarded instead) and T will unwind and is able to process another incoming message.

This can done in a new protocol (NON_BLOCKING_SENDS), placed directly above the transport:

• In up(Message) or up(MessageBatch), the current thread T is added to a list of threads. This is the list of threads currently processing received (incoming) messages
• When up() returns, the current thread T is again removed from the list
• On a down(Message), we check if the current thread is in the list. If yes, this means that the incoming thread is sending a message, and therefore, we mark the message as DONT_BLOCK.

While we can look for all JGroups (protocols) that send an (internal) message when processing a received message, and mark them as DONT_BLOCK, this protocol can be used to also detect when the application sends a message in a receive() callback.

Therefore the use case for NON_BLOCKING_SENDS is to tag application messages sent on an inciming thread with DONT_BLOCK. JGroups-internal use will already use DONT_BLOCK, so the protocol is not needed there...

This JIRA makes only sense for TCP: both TCP_NIO2 and UDP don't block when sending a message, therefore the TransferQueueBundler's queue will never be full.