Under heavy load (900 users) I see contention on TimeoutFactory due to the methods TransactionFactory.dropTimeout and newTimeout.
Here is the method:

private boolean dropTimeout(TimeoutImpl timeout)
{
synchronized (this)
{
if (timeout.index > 0)

{ // Active timeout, remove it. // INV: assertExpr(q[timeout.index] == timeout); // INV: checkTree(); removeNode(timeout.index); // INV: checkTree(); timeout.index = TimeoutImpl.DONE; timeout.nextFree = freeList; freeList = timeout; // INV: checkFreeList(); // execution cancelled return true; }

else

{ // has already been executed (DONE) or // is currently executing (TIMEOUT) return false; }

}
}

Stack Trace below:

"SocketServerInvokerThread-10.64.36.1-133" id=241 idx=0x1d4 tid=16291 prio=5 alive, in native, blocked – Blocked trying to get lock: org/jboss/util/timeout/TimeoutFactory@0x2000000015c53290[fat lock] at jrockit/vm/Threads.waitForSignal()V(Native Method) at jrockit/vm/Locks.fatLockBlockOrSpin(JLjrockit/vm/ObjectMonitor;II)V(Unknown Source)[optimized] at jrockit/vm/Locks.lockFat(Ljava/lang/Object;JLjrockit/vm/ObjectMonitor;Z)Ljava/lang/Object;(Unknown Source)[optimized] at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)[optimized] at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized] at org/jboss/util/timeout/TimeoutFactory.newTimeout(JLorg/jboss/util/timeout/TimeoutTarget;)Lorg/jboss/util/timeout/Timeout;(TimeoutFactory.java:515) at org/jboss/util/timeout/TimeoutFactory.schedule(JLorg/jboss/util/timeout/TimeoutTarget;)Lorg/jboss/util/timeout/Timeout;(TimeoutFactory.java:294) at org/jboss/util/timeout/TimeoutFactory.createTimeout(JLorg/jboss/util/timeout/TimeoutTarget;)Lorg/jboss/util/timeout/Timeout;(TimeoutFactory.java:246) at org/jboss/tm/TransactionImpl.<init>(J)V(TransactionImpl.java:220) at org/jboss/tm/TxManager.begin()V(TxManager.java:213) at org/jboss/ejb/plugins/TxInterceptorCMT.startTransaction(Lorg/jboss/invocation/Invocation;)I(TxInterceptorCMT.java:476) at org/jboss/ejb/plugins/TxInterceptorCMT.runWithTransactions(Lorg/jboss/invocation/Invocation;)Ljava/lang/Object;(TxInterceptorCMT.java:329)[optimized] at org/jboss/ejb/plugins/TxInterceptorCMT.invoke(Lorg/jboss/invocation/Invocation;)Ljava/lang/Object;(TxInterceptorCMT.java:181)[optimized] at org/jboss/ejb/plugins/LogInterceptor.invoke(Lorg/jboss/invocation/Invocation;)Ljava/lang/Object;(LogInterceptor.java:165)[optimized] at org/jboss/ejb/plugins/CleanShutdownInterceptor.invoke(Lorg/jboss/invocation/Invocation;)Ljava/lang/Object;(CleanShutdownInterceptor.java:278) at org/jboss/ejb/plugins/ProxyFactoryFinderInterceptor.invoke(Lorg/jboss/invocation/Invocation;)Ljava/lang/Object;(ProxyFactoryFinderInterceptor.java:136)[optimized] at jrockit/reflect/CompiledMethodInvoker.invoke0(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source) at jrockit/reflect/CompiledMethodInvoker.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)[optimized] at java/lang/reflect/Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;J)Ljava/lang/Object;(Unknown Source)[optimized] at org/jboss/mx/interceptor/ReflectedDispatcher.invoke(Lorg/jboss/mx/server/Invocation;)Ljava/lang/Object;(ReflectedDispatcher.java:155) at org/jboss/mx/server/AbstractMBeanInvoker.invoke(Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object;(AbstractMBeanInvoker.java:189)[optimized] at org/jboss/mx/server/MBeanServerImpl.invoke(Ljavax/management/ObjectName;Ljava/lang/String;[Ljava/lang/Object;[Ljava/lang/String;)Ljava/lang/Object;(MBeanServerImpl.java:659) at org/jboss/invocation/unified/server/UnifiedInvokerHA.invoke(Lorg/jboss/remoting/InvocationRequest;)Ljava/lang/Object;(UnifiedInvokerHA.java:146) at org/jboss/remoting/ServerInvoker.invoke(Lorg/jboss/remoting/InvocationRequest;)Ljava/lang/Object;(ServerInvoker.java:828) at org/jboss/remoting/ServerInvoker.invoke(Ljava/lang/Object;)Ljava/lang/Object;(ServerInvoker.java:681) at org/jboss/remoting/transport/socket/ServerThread.processInvocation()V(ServerThread.java:358)