When we send a message with DONT_BLOCK set, it might get dropped and not retransmitted in the following scenario (unicast message, but the same issue exists for multicast messages):

• A sends messages M1, M2 and M3 to B (each message is sent by a different thread)
• M2 has flag DONT_BLOCK set
• The send buffer to B is full
• M1 blocks
• M2 is discarded: the send buffer is full and DONT_BLOCK is set
• M3 blocks

When ACKs are received, causing the send buffer to unblock, M1 and M3 are sent. However, M2 will never be sent as it is not retransmitted (not in the send buffer).

The problem is that we skipped a seqno for M2: this causes M1 to be delivered at B, but not M2 and M3. Subsequent messages to B will pile up in B's receive buffer for A and eventually cause OOME.

Fix

• Ignore DONT_BLOCK flag and block anyway
• If we want a message to always bypass blocking in UNICAST4 and NAKACK4, use flag NO_RELIABILITY: this will bypass the above 2 protocols, but at the cost of not being reliable.