[Email from David Forget]

Hi,

We found some issue using JGroups 2.2.9 in TCP mode. With a cluster of 10 nodes with one node that only connect and disconnect to the group every minutes. We observer in this scenario using Thread Dumps that some members have a Sender thread leak (10 receivers threads, 325 Senders threads).

Note: JGroups 2.2.8 does not have the same problem.

Here is a piece of code from the Connection.Sender inner class in the ConnectionTable.java file.

void stop() {

if(senderThread != null)

{ senderThread.interrupt(); senderThread=null; running=false; }

}

public void run() {

byte[] data;

while(senderThread != null && senderThread.equals(Thread.currentThread())) {

try

{ data=(byte[])send_queue.remove(); if(data == null) continue; _send(data, 0, data.length); }

catch(QueueClosedException e)

{ break; }

}

The problem with this code is that when the stop() method is called, it is possible that the thread exits from the send_queue.remove() with null. If this thread executes, loops and blocks on the remove method again before the senderThread is put to null, the thread will be blocked forever (Note that the interrupt flag of the thread is reseted after throwing an InterruptedException so that the next wait() will not exit). Because of the Reaper thread, this can cause a severe memory leak and crash the application.

The fix we suggest is to modify the stop() method of the Sender class and add a call to send_queue.close(). This way, even if the Sender thread loops and re-executes the send_queue.remove() method, the remove() method will throw a QueueClosedException and the thread will terminate properly. We have tested this fix and it successfully prevents the Sender thread leak.

void stop() {

if(senderThread != null)

{ send_queue.close(false); senderThread.interrupt(); senderThread=null; running=false; }

}

Regards,

David Forget