Details
-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
3.3.1.Final
Description
We encountered a deadlock in the remoting3 when using the JMX monitoring to also dump connection state. See the thread dump information below. We are calling the JMX monitor operation to dumpConnectionStateToString which causes a deadlock with the read thread responding to remoting3 requests. This caused one of our nodes to stop handling remote EJB requests entirely.
Found one Java-level deadlock:
=============================
"pool-2-thread-12622":
waiting to lock monitor 0x00007f771401aa58 (object 0x0000000702646438, a java.util.ArrayDeque),
which is held by "Remoting "XXXXX.com" read-1"
"Remoting "XXXXX.com" read-1":
waiting to lock monitor 0x00007f76ec1d1f78 (object 0x000000071e4b4318, a java.util.Collections$SynchronizedSet),
which is held by "pool-2-thread-12622"
Java stack information for the threads listed above:
===================================================
"pool-2-thread-12622":
at org.jboss.remoting3.remote.RemoteConnectionHandler.dumpState(RemoteConnectionHandler.java:430)
- waiting to lock <0x0000000702646438> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnectionProvider.doGetConnectionState(RemoteConnectionProvider.java:141)
- locked <0x000000071e4b4318> (a java.util.Collections$SynchronizedSet)
at org.jboss.remoting3.remote.RemoteConnectionProvider.doGetConnectionState(RemoteConnectionProvider.java:148)
at org.jboss.remoting3.remote.RemoteConnectionProvider.access$100(RemoteConnectionProvider.java:76)
at org.jboss.remoting3.remote.RemoteConnectionProvider$1.dumpConnectionStateToString(RemoteConnectionProvider.java:121)
at sun.reflect.GeneratedMethodAccessor2882.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor408.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:1455)
at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:731)
at org.jboss.remotingjmx.protocol.v2.ServerProxy$InvokeHandler.handle(ServerProxy.java:952)
at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1$1.run(ServerCommon.java:153)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:75)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor$1.run(ServerInterceptorFactory.java:70)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
at org.jboss.as.jmx.ServerInterceptorFactory$Interceptor.handleEvent(ServerInterceptorFactory.java:70)
at org.jboss.remotingjmx.protocol.v2.ServerCommon$MessageReciever$1.run(ServerCommon.java:149)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"Remoting "XXXXX.com" read-1":
at java.util.Collections$SynchronizedCollection.remove(Collections.java:1639)
- waiting to lock <0x000000071e4b4318> (a java.util.Collections$SynchronizedSet)
at org.jboss.remoting3.remote.RemoteConnectionProvider.removeConnectionHandler(RemoteConnectionProvider.java:275)
at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:117)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:81)
- locked <0x0000000702646438> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.nio.NioHandle.run(NioHandle.java:90)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)
Found 1 deadlock.
