Uploaded image for project: 'JBoss Remoting (3+)'
  1. JBoss Remoting (3+)
  2. REM3-203

Deadlock between remote read and JMX threads

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 3.3.1.Final
    • Hide

      See stack trace. Startup a Jboss node with some basic EJB deployed. Remotely perform calls to the EJB node. Concurrently, request the JMX operation dumpConnectionStateToString.

      Show
      See stack trace. Startup a Jboss node with some basic EJB deployed. Remotely perform calls to the EJB node. Concurrently, request the JMX operation dumpConnectionStateToString.

      We encountered a deadlock in the remoting3 when using the JMX monitoring to also dump connection state. See the thread dump information below. We are calling the JMX monitor operation to dumpConnectionStateToString which causes a deadlock with the read thread responding to remoting3 requests. This caused one of our nodes to stop handling remote EJB requests entirely.

      Found one Java-level deadlock:
      =============================
      "pool-2-thread-12622":
        waiting to lock monitor 0x00007f771401aa58 (object 0x0000000702646438, a java.util.ArrayDeque),
        which is held by "Remoting "XXXXX.com" read-1"
      "Remoting "XXXXX.com" read-1":
        waiting to lock monitor 0x00007f76ec1d1f78 (object 0x000000071e4b4318, a java.util.Collections$SynchronizedSet),
        which is held by "pool-2-thread-12622"
      
      
      Java stack information for the threads listed above:
      ===================================================
      "pool-2-thread-12622":
              at org.jboss.remoting3.remote.RemoteConnectionHandler.dumpState(RemoteConnectionHandler.java:430)
              - waiting to lock <0x0000000702646438> (a java.util.ArrayDeque)
              at org.jboss.remoting3.remote.RemoteConnectionProvider.doGetConnectionState(RemoteConnectionProvider.java:141)
              - locked <0x000000071e4b4318> (a java.util.Collections$SynchronizedSet)
              at org.jboss.remoting3.remote.RemoteConnectionProvider.doGetConnectionState(RemoteConnectionProvider.java:148)
              at org.jboss.remoting3.remote.RemoteConnectionProvider.access$100(RemoteConnectionProvider.java:76)
              at org.jboss.remoting3.remote.RemoteConnectionProvider$1.dumpConnectionStateToString(RemoteConnectionProvider.java:121)
              at sun.reflect.GeneratedMethodAccessor2882.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
              at sun.reflect.GeneratedMethodAccessor408.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
              at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
              at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
              at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
              at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
              at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
              at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
              at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
              at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
              at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
              at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:1455)
              at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:731)
              at org.jboss.remotingjmx.protocol.v2.ServerProxy$InvokeHandler.handle(ServerProxy.java:952)
              at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1$1.run(ServerCommon.java:153)
              at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:75)
              at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:70)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:415)
              at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
              at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor.handleEvent(ServerInterceptorFactory.java:70)
              at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:149)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:745)
      
      
      "Remoting "XXXXX.com" read-1":
              at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639)
              - waiting to lock <0x000000071e4b4318> (a java.util.Collections$SynchronizedSet)
              at org.jboss.remoting3.remote.RemoteConnectionProvider.removeConnectionHandler(RemoteConnectionProvider.java:275)
              at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:117)
              at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
              - locked <0x0000000702646438> (a java.util.ArrayDeque)
              at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
              at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
              at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
              at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
              at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
              at org.xnio.nio.NioHandle.run(NioHandle.java:90)
              at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
      
      Found 1 deadlock.
      

              Unassigned Unassigned
              arothian_jira Kevin Formsma (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: