-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
4.0.7.Final, 4.0.8.Final, 4.0.21.Final
In our application running with Wildfly8.2.1, we are facing below deadlock issue again and again after couple of days:
When the connection is closed while we are sending some content we have a deadlock happening between the RemoteConnection.RemoteWriteListener.queue and the BufferPipeOutputStream
As per the below link the issue has been fixed in 4.0.21 remoting jar:
https://issues.jboss.org/browse/REM3-204
but the code has been reverted in jboss-remoting 4.0.11 version due to some other issue as mentioned in the below link:
https://bugzilla.redhat.com/show_bug.cgi?id=1262114
Also the same deadlock is occuring in higher version of wildfly.
Java stack information for the threads with deadlock:
===================================================
Found one Java-level deadlock:
=============================
"pool-3-thread-39457":
waiting to lock monitor 0x00007f13947178f8 (object 0x0000000567801c98, a java.util.ArrayDeque),
which is held by "XNIO-1 I/O-1"
"XNIO-1 I/O-1":
waiting to lock monitor 0x00007f1205f31e88 (object 0x0000000568c09f00, a org.xnio.streams.BufferPipeOutputStream),
which is held by "pool-3-thread-39457"
Java stack information for the threads listed above:
===================================================
"pool-3-thread-39457":
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:294)
- waiting to lock <0x0000000567801c98> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:122)
at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:154)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:122)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:115)
at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:139) - locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:157) - locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
at org.jboss.remoting3.remote.OutboundMessage.close(OutboundMessage.java:283) - locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
at org.xnio.IoUtils.safeClose(IoUtils.java:134)
at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:187)
at org.jboss.remotingjmx.protocol.v2.ServerProxy.writeResponse(ServerProxy.java:319)
at org.jboss.remotingjmx.protocol.v2.ServerProxy.access$3400(ServerProxy.java:116)
at org.jboss.remotingjmx.protocol.v2.ServerProxy$GetAttributeHandler.handle(ServerProxy.java:693)
at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1$1.run(ServerCommon.java:153)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:75)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor.handleEvent(ServerInterceptorFactory.java:70)
at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:149)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"XNIO-1I/O-1":
at org.jboss.remoting3.remote.OutboundMessage.cancel(OutboundMessage.java:288) - waiting to lock <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeMessages(RemoteConnectionChannel.java:560)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:542)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:429)
at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:115)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81) - locked <0x0000000567801c98> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:199)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:113)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
Found 1 deadlock.
So could anyone has any solution for the deadlock.