Uploaded image for project: 'JBoss Remoting (3+)'
  1. JBoss Remoting (3+)
  2. REM3-327

Deadlock in remoting (Deadlock when connection is closing while we are writing)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 4.0.7.Final, 4.0.8.Final, 4.0.21.Final

      In our application running with Wildfly8.2.1, we are facing below deadlock issue again and again after couple of days:

      When the connection is closed while we are sending some content we have a deadlock happening between the RemoteConnection.RemoteWriteListener.queue and the BufferPipeOutputStream

      As per the below link the issue has been fixed in 4.0.21 remoting jar:
      https://issues.jboss.org/browse/REM3-204

      but the code has been reverted in jboss-remoting 4.0.11 version due to some other issue as mentioned in the below link:
      https://bugzilla.redhat.com/show_bug.cgi?id=1262114

      Also the same deadlock is occuring in higher version of wildfly.

      Java stack information for the threads with deadlock:
      ===================================================
      Found one Java-level deadlock:
      =============================
      "pool-3-thread-39457":
      waiting to lock monitor 0x00007f13947178f8 (object 0x0000000567801c98, a java.util.ArrayDeque),
      which is held by "XNIO-1 I/O-1"
      "XNIO-1 I/O-1":
      waiting to lock monitor 0x00007f1205f31e88 (object 0x0000000568c09f00, a org.xnio.streams.BufferPipeOutputStream),
      which is held by "pool-3-thread-39457"

      Java stack information for the threads listed above:
      ===================================================
      "pool-3-thread-39457":
      at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:294)

      • waiting to lock <0x0000000567801c98> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:122)
        at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:154)
        at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:122)
        at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:115)
        at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:139)
      • locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
        at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:157)
      • locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
        at org.jboss.remoting3.remote.OutboundMessage.close(OutboundMessage.java:283)
      • locked <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
        at java.io.FilterOutputStream.close(FilterOutputStream.java:159)
        at org.xnio.IoUtils.safeClose(IoUtils.java:134)
        at org.jboss.remotingjmx.protocol.v2.Common.write(Common.java:187)
        at org.jboss.remotingjmx.protocol.v2.ServerProxy.writeResponse(ServerProxy.java:319)
        at org.jboss.remotingjmx.protocol.v2.ServerProxy.access$3400(ServerProxy.java:116)
        at org.jboss.remotingjmx.protocol.v2.ServerProxy$GetAttributeHandler.handle(ServerProxy.java:693)
        at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1$1.run(ServerCommon.java:153)
        at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:75)
        at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:70)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
        at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor.handleEvent(ServerInterceptorFactory.java:70)
        at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:149)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        "XNIO-1 I/O-1":
        at org.jboss.remoting3.remote.OutboundMessage.cancel(OutboundMessage.java:288)
      • waiting to lock <0x0000000568c09f00> (a org.xnio.streams.BufferPipeOutputStream)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeMessages(RemoteConnectionChannel.java:560)
        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:542)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:429)
        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:115)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
      • locked <0x0000000567801c98> (a java.util.ArrayDeque)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:199)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:113)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)

      Found 1 deadlock.

      So could anyone has any solution for the deadlock.

              Unassigned Unassigned
              NeerajGupta1234 Neeraj Gupta (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: