-
Bug
-
Resolution: Done
-
Critical
-
JBossAS-3.2.7 Final, JBossAS-4.0.1 Final, JBossAS-5.0.0.Beta1
-
None
-
None
The HA-JNDI write operations are using the java synchronized modifier to attempt to make the write operation atomtic. This has a deadlock problem due to the fact that the TreeHead method encapsulates the NamingServer write operation wich is already synchronized, with the jndi group rpc to replicate the change to the cluster. This leads to a trival deadlock scenario where the replication rpc times out due to another write operation waiting for the TreeHead mutex:
"RMI TCP Connection(11)-192.168.2.100" daemon prio=5 tid=0x00a3c410 nid=0x4bd0 in Object.wait() [5bbf000..5bbfdb4]
at java.lang.Object.wait(Native Method)
at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:505)
at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:183)
- locked <0x153748c8> (a java.lang.Object)
at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:384)
at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:134)
at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:620)
at org.jboss.ha.jndi.TreeHead.rebind(TreeHead.java:133) - locked <0x14460060> (a org.jboss.ha.jndi.TreeHead)
at org.jboss.ha.jndi.HAJNDI.rebind(HAJNDI.java:119) - locked <0x1445f610> (a org.jboss.ha.jndi.HAJNDI)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ha.framework.server.HARMIServerImpl.invoke(HARMIServerImpl.java:221)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:261)
at sun.rmi.transport.Transport$1.run(Transport.java:148)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:144)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:460)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:701)
at java.lang.Thread.run(Thread.java:534)
"MessageDispatcher up processing thread" daemon prio=5 tid=0x033379d8 nid=0x7ea4
waiting for monitor entry [573f000..573fdb4]
at org.jboss.ha.jndi.TreeHead._rebind(TreeHead.java:115)
- waiting to lock <0x14460060> (a org.jboss.ha.jndi.TreeHead)
at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:236)
at org.jboss.ha.framework.server.HAPartitionImpl.handle(HAPartitionImpl.java:828)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:615)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:512)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:722)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:554)
at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:691)
at java.lang.Thread.run(Thread.java:534)
The only state in TreeHead needing atomic access is the inheritted NamingServer state and this is already atomic. The synchronization on the TreeHead methods needs to be removed. There is no guarentee with respect to cluster wide writes that may be involve the same node. The existing HA-JNDI service was designed with read-only federation of the unclustered JNDI trees in the cluster in mind.