Okay, I know what the problem is.

When a multicast message is received from sender A, then a lock is acquired for A (or we block until A's done processing the message and releases the lock).

When the receive() method would send more messages down the stack, that lock might get held for potentially a long time. So I made the following change: I assume that calling down() from within receive() means that the message has been delivered, so I release the lock held for A and someone else can now acquire that lock. If you comment the lines of ProtocolStack.down() and recompile JGroups, then you will always see a count of 1:

public Object down(Event evt) {
ReentrantLock lock=locks.remove(Thread.currentThread());
//if(lock != null && lock.isHeldByCurrentThread())

{ // lock.unlock(); //if(log.isTraceEnabled()) // log.trace("released lock held by " + Thread.currentThread()); //}

if(top_prot != null)
return top_prot.down(evt);
return null;
}

The issue here is that I don't want to block incoming calls just because the receive() method sends other messages. This is described in http://jira.jboss.com/jira/browse/JGRP-535.

I might introduce a flag in ProtocolStack to configure this behavior...