Uploaded image for project: 'JBoss Enterprise Application Platform 4 and 5'
  1. JBoss Enterprise Application Platform 4 and 5
  2. JBPAPP-9946

JBossCache: TimeoutException after app undeploy, Oracle JDK 6

XMLWordPrintable

    • Not Required
    • NEW

      After application undeploy in a cluster, we are seeing the following exception.
      There are few of them in a row (3 or so), observed in the following cache modes: both buddy and total, in synchronized mode. Did not try in async. 

      javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /_BUDDY_BACKUP_/10.16.90.56_55200/sfsb could not be acquired by Thread[Incoming-10,10.16.90.54:55200,10,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=1, waitingWriters=0, waitingUpgrader=0)
	at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:77)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:52)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:182)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:569)
	at org.jboss.ejb3.session.InvokableContextClassProxyHack._dynamicInvoke(InvokableContextClassProxyHack.java:53)
	at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:91)
	at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
	at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
Caused by: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /_BUDDY_BACKUP_/10.16.90.56_55200/sfsb could not be acquired by Thread[Incoming-10,10.16.90.54:55200,10,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=1, waitingWriters=0, waitingUpgrader=0)
	at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:332)
	at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:276)
	at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:499)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.acquireNodeLock(PessimisticNodeBasedLockManager.java:286)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.lock(PessimisticNodeBasedLockManager.java:186)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.lockPessimistically(PessimisticNodeBasedLockManager.java:98)
	at org.jboss.cache.interceptors.PessimisticLockInterceptor.handleGetNodeCommand(PessimisticLockInterceptor.java:360)
	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitGetNodeCommand(PrePostProcessingCommandInterceptor.java:293)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGetNodeCommand(AbstractVisitor.java:115)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGetNodeCommand(AbstractVisitor.java:115)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
	at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
	at org.jboss.cache.commands.AbstractVisitor.visitGetNodeCommand(AbstractVisitor.java:115)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGetNodeCommand(AbstractVisitor.java:115)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
	at org.jboss.cache.commands.AbstractVisitor.visitGetNodeCommand(AbstractVisitor.java:115)
	at org.jboss.cache.commands.read.GetNodeCommand.acceptVisitor(GetNodeCommand.java:76)
	at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:288)
	at org.jboss.cache.invocation.CacheInvocationDelegate.getNode(CacheInvocationDelegate.java:494)
	at org.jboss.cache.commands.read.GravitateDataCommand.perform(GravitateDataCommand.java:157)
	at org.jboss.cache.interceptors.CallInterceptor.invokeCommand(CallInterceptor.java:108)
	at org.jboss.cache.interceptors.CallInterceptor.handleDefault(CallInterceptor.java:99)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.handleGravitateDataCommand(PrePostProcessingCommandInterceptor.java:375)
	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitGravitateDataCommand(PrePostProcessingCommandInterceptor.java:365)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
	at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:130)
	at org.jboss.cache.commands.AbstractVisitor.visitGravitateDataCommand(AbstractVisitor.java:135)
	at org.jboss.cache.commands.read.GravitateDataCommand.acceptVisitor(GravitateDataCommand.java:240)
	at org.jboss.cache.interceptors.InterceptorChain.invoke(InterceptorChain.java:288)
	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:300)
	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:246)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
	at org.jgroups.JChannel.up(JChannel.java:1336)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
	at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:432)
	at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
	at org.jgroups.protocols.FC.up(FC.java:473)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
	at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
	at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:628)
	at org.jgroups.protocols.UNICAST.up(UNICAST.java:309)
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:750)
	at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:174)
	at org.jgroups.protocols.FD.up(FD.java:284)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
	at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
	at org.jgroups.protocols.Discovery.up(Discovery.java:264)
	at org.jgroups.protocols.PING.up(PING.java:273)
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
	at org.jgroups.protocols.TP.access$100(TP.java:50)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:218)
	at org.jboss.remoting.Client.invoke(Client.java:2084)
	at org.jboss.remoting.Client.invoke(Client.java:879)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
	at $Proxy9.invoke(Unknown Source)
	at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
	at $Proxy8.getSerial(Unknown Source)
	at org.jboss.smartfrog.ejb3.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:94)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
	at java.lang.Thread.run(Thread.java:662)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
	at $Proxy9.invoke(Unknown Source)
	at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
	at $Proxy8.getSerial(Unknown Source)
	at org.jboss.smartfrog.ejb3.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:94)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
	at java.lang.Thread.run(Thread.java:662)

      javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /_BUDDY_BACKUP_/10.16.90.56_55200/sfsb/ear=ejb3-session-test-ear-1.0.0.ear,jar=ejb3-session-test-ejb.jar,name=SerialSessionBean,service=EJB3 could not be acquired by Thread[Incoming-18,10.16.90.54:55200,10,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0

      Here is the exception when using total replication:

      javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /sfsb/ear=ejb3-session-test-ear-1.0.0.ear,jar=ejb3-session-test-ejb.jar,name=SerialSessionBean,service=EJB3 could not be acquired by Thread[Incoming-16,10.16.90.58:55200,5,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
	at org.jboss.ejb3.tx.Ejb3TxPolicy.handleExceptionInOurTx(Ejb3TxPolicy.java:77)
	at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:83)
	at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:52)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:182)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:569)
	at org.jboss.ejb3.session.InvokableContextClassProxyHack._dynamicInvoke(InvokableContextClassProxyHack.java:53)
	at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:91)
	at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
	at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
	at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
	at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
	at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
	at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
Caused by: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /sfsb/ear=ejb3-session-test-ear-1.0.0.ear,jar=ejb3-session-test-ejb.jar,name=SerialSessionBean,service=EJB3 could not be acquired by Thread[Incoming-16,10.16.90.58:55200,5,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)
	at org.jboss.cache.lock.IdentityLock.acquireReadLock0(IdentityLock.java:332)
	at org.jboss.cache.lock.IdentityLock.acquireReadLock(IdentityLock.java:276)
	at org.jboss.cache.lock.IdentityLock.acquire(IdentityLock.java:499)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.acquireNodeLock(PessimisticNodeBasedLockManager.java:286)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.lock(PessimisticNodeBasedLockManager.java:186)
	at org.jboss.cache.lock.PessimisticNodeBasedLockManager.lockPessimistically(PessimisticNodeBasedLockManager.java:98)
	at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutCommand(PessimisticLockInterceptor.java:129)
	at org.jboss.cache.interceptors.PessimisticLockInterceptor.handlePutKeyValueCommand(PessimisticLockInterceptor.java:100)
	at org.jboss.cache.interceptors.base.PrePostProcessingCommandInterceptor.visitPutKeyValueCommand(PrePostProcessingCommandInterceptor.java:88)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:150)
	at org.jboss.cache.interceptors.ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:107)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:131)
	at org.jboss.cache.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)
	at org.jboss.cache.interceptors.TxInterceptor.handleDefault(TxInterceptor.java:283)
	at org.jboss.cache.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:119)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)
	at org.jboss.cache.interceptors.InvocationContextInterceptor.visitPutKeyValueCommand(InvocationContextInterceptor.java:82)
	at org.jboss.cache.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)
	at org.jboss.cache.interceptors.InterceptorChain.invokeRemote(InterceptorChain.java:325)
	at org.jboss.cache.commands.remote.ReplicateCommand.processSingleCommand(ReplicateCommand.java:139)
	at org.jboss.cache.commands.remote.ReplicateCommand.perform(ReplicateCommand.java:115)
	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:319)
	at org.jboss.cache.marshall.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:246)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
	at org.jgroups.JChannel.up(JChannel.java:1336)
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
	at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:494)
	at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)
	at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
	at org.jgroups.protocols.FC.up(FC.java:473)
	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
	at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
	at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
	at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:174)
	at org.jgroups.protocols.FD.up(FD.java:284)
	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
	at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
	at org.jgroups.protocols.Discovery.up(Discovery.java:264)
	at org.jgroups.protocols.PING.up(PING.java:273)
	at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
	at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
	at org.jgroups.protocols.TP.access$100(TP.java:50)
	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
	at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:218)
	at org.jboss.remoting.Client.invoke(Client.java:2084)
	at org.jboss.remoting.Client.invoke(Client.java:879)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
	at $Proxy8.invoke(Unknown Source)
	at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
	at $Proxy7.getSerial(Unknown Source)
	at org.jboss.smartfrog.ejb3.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:94)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
	at java.lang.Thread.run(Thread.java:662)
	at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:122)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
	at $Proxy8.invoke(Unknown Source)
	at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:188)
	at $Proxy7.getSerial(Unknown Source)
	at org.jboss.smartfrog.ejb3.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:94)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:51)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:87)
	at java.lang.Thread.run(Thread.java:662)

      javax.ejb.EJBException: java.lang.RuntimeException: org.jboss.cache.lock.TimeoutException read lock for /sfsb/ear=ejb3-session-test-ear-1.0.0.ear,jar=ejb3-session-test-ejb.jar,name=SerialSessionBean,service=EJB3 could not be acquired by Thread[Incoming-10,10.16.90.56:55200,10,Thread Pools] after 15000 ms. Locks: Read lock owners: []
Write lock owner: Thread[HDScanner,5,jboss]
, lock info: write owner=Thread[HDScanner,5,jboss] (activeReaders=0, activeWriter=Thread[HDScanner,5,jboss], waitingReaders=0, waitingWriters=0, waitingUpgrader=0)

        1. TRACE-LOG.zip
          169 kB

              chaowan@redhat.com Chao Wang
              jkudrnac@redhat.com Jitka Kozana
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: