When NAKACK receives a message from C, it gets the NakReceiverWindow for C and acquires a lock on C, and then passes up the message. When the call returns, the lock is received (it is actually currently a synchronized(win) statement).

The reason is that this is needed to deliver all messages from C in the order in which they were sent.

The issue is that, while a message from C is delivered, all other threads are blocked on the lock for C's NakReceiverWindow. Especially the case when an up() method calls down() before returning, and down() blocks, this increases the duration during which the lock is kept. Example:

• Message from C acquires a lock on C's NakReceiverWindow
• Message is passed up
• The application gets the receive() callback
• Inside the receive(), the application calls Channel.send() which results in a down()
• Only when the down() placed the message on the network and returned will the up() method unwind its stack and return

--> Result: we kept a lock on C's NRW for much longer than we needed to. The lock should be kept until C's message has been DELIVERED, but not until the down() method returns !
--> If FC.down() blocks for credits, this makes it even worse !

SOLUTION:

• Investigate associating the lock acquired in NAKACK for C's NakReceiverWindow with the thread local. When the message is delivered (semantics ?), release the lock, this is sooner than when the up() method returns in NAKACK.
• This could also be done for example in STABLE.handleRegularMessage(), where we send the message up first, the release the lock and only then multicast the STABLE message

Investigate UNICAST as well.