-
Bug
-
Resolution: Done
-
Blocker
-
5.2.0.Final
-
None
It looks like the Map/Reduce intermediate caches use pessimistic transactions, but the transactions are not guaranteed to write to the keys in the same order. So it's possible for two tasks to get into a deadlock, ending with a TimeoutException:
16:18:40,649 ERROR (testng-DistributedFourNodesMapReduceTest:) [UnitTestTestNGListener] Test testCombinerDoesNotChangeResult(org.infinispan.distexec.mapreduce.DistributedFourNodesMapReduceTest) failed. org.infinispan.CacheException: Could not invoke map phase of MapReduce task on remote nodes at org.infinispan.distexec.mapreduce.MapReduceTask.invokeEverywhere(MapReduceTask.java:562) at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhase(MapReduceTask.java:374) at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:315) at org.infinispan.distexec.mapreduce.BaseWordCountMapReduceTest.testCombinerDoesNotChangeResult(BaseWordCountMapReduceTest.java:188) ... Caused by: org.infinispan.CacheException: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task 04244b4b-08b1-4fc4-9755-ed02f3f35a3a at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:97) at org.infinispan.commands.read.MapCombineCommand.perform(MapCombineCommand.java:89) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:110) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:82) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:244) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:217) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483) ... Caused by: org.infinispan.CacheException: Could not move intermediate keys/values for M/R task 04244b4b-08b1-4fc4-9755-ed02f3f35a3a at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.combine(MapReduceManagerImpl.java:281) at org.infinispan.distexec.mapreduce.MapReduceManagerImpl.mapAndCombineForDistributedReduction(MapReduceManagerImpl.java:95) ... 26 more Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [JBoss] for requestor [GlobalTransaction:<NodeD-56763>:10429:remote]! Lock held by [GlobalTransaction:<NodeB-55590>:10432:remote] at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:217) at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:190) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:190) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:125) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:248) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:131) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) at org.infinispan.commands.AbstractVisitor.visitLockControlCommand(AbstractVisitor.java:177) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:131) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:125) at org.infinispan.interceptors.TxInterceptor.visitLockControlCommand(TxInterceptor.java:174) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:131) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:212) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:187) at org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:131) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:131) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:129) at org.infinispan.interceptors.InvocationContextInterceptor.visitLockControlCommand(InvocationContextInterceptor.java:98) at org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:131) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:347) at org.infinispan.commands.control.LockControlCommand.perform(LockControlCommand.java:150) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95) at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:110) at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:82) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:244) at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:217) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:483) ...
- is blocked by
-
ISPN-2410 A command forwarded back to the originator can time out waiting on a key already locked by itself
- Closed