-
Bug
-
Resolution: Done
-
Major
-
2.6, 2.7
-
None
If Thread#interrupt() is invoked on a thread that is blocked in an invocation of the wait(), wait(long), or wait(long, int) methods of the Object class, or of the join(), join(long), join(long, int), sleep(long), or sleep(long, int), methods of this class, then its interrupt status will be cleared and it will receive an InterruptedException. (citing Thread#interrupt javadoc)
Conventional wisdom is that interrupt signal was always cleared. However, in cases except the above interrupt signal is actually not cleared! JGroups code that is throwing InterruptedException should make sure that it also clears the interrupt status. Otherwise, if we wrongly assume that interrupt status is always cleared, we can have inconsistencies that can lead to hard-to-trace bugs (See below). We should verify that throughout our code that we always clear interrupt status.
Gray said on javagroups-list:
We have been seeing a large percentage of our production servers (on 2.6.1)
spinning above 100% of CPU. After some investigation, we seem to have found
an infinite loop in Scheduler.java. If sched_thread gets interrupted, its
interrupt flag never gets reset so it spins forever suspending and resuming
the current_thread and throwing and catching InterruptedException(s).
According to jconsole, one of our boxes has done this 690 million times –
nice effort. When this is happening, the memory space is alternating
between 200 and 300 mb which makes sense given the exception bandwidth.
We are running our RPC dispatchers wth deadlock_detection and
concurrent_processing set to true and use_concurrent_stack="true".
Here are http://www.nabble.com/file/p15395455/stack_dumps.txt some stack
snapshots of the spinning thread . As you can see they are all in
Scheduler.run. Here is the section from
http://www.nabble.com/file/p15395455/Scheduler.java.txt Scheduler.java with
the offending part of the code including where sched_thread is being
interrupted by a call to Scheduler.addPrio(). The pluses show where we
think a missing Thread.interrupt() call should be.
Here's the http://www.nabble.com/file/p15395455/Scheduler.java.patch.txt
patch for Scheduler.java (from 2.6.1) which adds in the interrupt call.
This is all with Jgroups 2.6.1. Anyone else see this behavior? We're going
to roll this patch out to see if it stops the spinning.
gray
- blocks
-
JGRP-654 Scheduler thread goes in infinite loop after getting interrupted by the priority task
- Closed