This seems like a bug in Scheduler class. The scheduler thread goes into infinite loop and never recovers after being interrupted by the priority task (upon deadlock detection). Here is what I think happens (with both deadlock_detection and concurrent_processing enabled for RpcDispatcher),

1) The tasks are added to the Scheduler Queue
2) If the deadlock_detection is turned on and there is a kind of
deadlock (2-way method calls) the RequestCorrelator.receiveMessage()
calls scheduler.addPrio(req) method
3) The addPrio(Runnable task) method adds the priority task to the
head of the queue and calls the interrupt method on the Scheduler
thread.
4) In the run() method of Scheduler we peek the queue, take the task
from head and run it
5) "sched_thread.isInterrupted()" in the run method returns true and
throws InterruptedException
6) Since, interrupted exception is thrown we bypass the
"queue.removeElement(current_task)" - and the task is not removed
from the queue.
7) The catch block of InterruptedException sets the
current_task.suspended=true, but never resets the interrupt flag
8) Now we keep going through the same loop for the same task in the
queue in the run() method as we never resets the interrupt flag.

I hope this is a correct diagnosis of this "TimeoutException" problem we have been seeing for last 2-3 months. I'm not sure if there are some nested Rpc calls in our system, but with deadlock-detection turned on they should be resolved. As tasks are never moved out of the queue, all the tasks just keep adding up (I see my queue over 2300 elements) and thus, we get TimeoutException for all the incoming calls.

I think we need to call "sched_thread.interrupted()" in the InterruptedException catch block so we can remove items from the queue as we process them. I've tried with this change (added Thread.interrupted at the end of InterruptedException catch block) in my local jgroups-all.jar and it seems to work fine for a long time now (3-4 hours, earlier it used to fail within 2 hours).

I haven't found any workaround for this problem if the deadlock detection logic is exercised.