2012-07-04 13:00:26,332 TRACE (OOB-61,ISPN,DistL1WriteSkewTest-NodeC-4119:___defaultcache) [org.infinispan.interceptors.InvocationContextInterceptor] Invoked with command VersionedPrepareCommand {modifications=[PutKeyValueCommand{key=counter, value=100, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, versionsSeen={counter=SimpleClusteredVersion{viewId=3, version=100}}, gtx=GlobalTransaction::17128:remote, cacheName='___defaultcache'} and InvocationContext [RemoteTxInvocationContext{flags=null}] *** sending prepare to NodeC and NodeD 2012-07-04 12:27:14,559 TRACE (Node-2:___defaultcache) [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] Replication task sending VersionedPrepareCommand {modifications=[PutKeyValueCommand{key=counter, value=19, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, versionsSeen={counter=SimpleClusteredVersion{viewId=3, version=19}}, gtx=GlobalTransaction::1568:local, cacheName='___defaultcache'} to addresses [DistL1WriteSkewTest-NodeC-4119, DistL1WriteSkewTest-NodeD-6770] with response mode GET_ALL *** executing prepare on NodeC 2012-07-04 12:27:14,694 TRACE (OOB-2,ISPN,DistL1WriteSkewTest-NodeC-4119:) [org.infinispan.transaction.TransactionTable] Created and registered remote transaction RemoteTransaction{modifications=[PutKeyValueCommand{key=counter, value=19, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], lookedUpEntries={}, lockedKeys= null, backupKeyLocks null, isMissingModifications false, tx=GlobalTransaction::1568:remote} 2012-07-04 12:27:14,711 TRACE (OOB-2,ISPN,DistL1WriteSkewTest-NodeC-4119:___defaultcache) [org.infinispan.transaction.RemoteTransaction] Adding key counter to tx GlobalTransaction::1568:remote *** rolling back transaction (I assume after timeout expired, nothing logged) 2012-07-04 12:27:29,662 TRACE (Node-2:) [org.infinispan.transaction.TransactionCoordinator] rollback transaction GlobalTransaction::1568:local 2012-07-04 12:27:29,664 TRACE (Node-2:___defaultcache) [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] Replication task sending RollbackCommand {gtx=GlobalTransaction::1568:local, cacheName='___defaultcache'} to addresses [DistL1WriteSkewTest-NodeC-4119, DistL1WriteSkewTest-NodeD-6770] with response mode GET_NONE *** rollback fails on NodeD 2012-07-04 12:27:30,191 TRACE (OOB-5,ISPN,DistL1WriteSkewTest-NodeD-6770:) [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] Attempting to execute command: RollbackCommand {gtx=GlobalTransaction::1568:local, cacheName='___defaultcache'} [sender=DistL1WriteSkewTest-NodeB-5373] 2012-07-04 12:27:30,191 TRACE (OOB-5,ISPN,DistL1WriteSkewTest-NodeD-6770:) [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] Did not find a RemoteTransaction for GlobalTransaction::1568:remote *** executing prepare on NodeD 2012-07-04 12:27:42,575 TRACE (OOB-1,ISPN,DistL1WriteSkewTest-NodeD-6770:) [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] Attempting to execute command: VersionedPrepareCommand {modifications=[PutKeyValueCommand{key=counter, value=19, flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1}], onePhaseCommit=false, versionsSeen={counter=SimpleClusteredVersion{viewId=3, version=19}}, gtx=GlobalTransaction::1568:local, cacheName='___defaultcache'} [sender=DistL1WriteSkewTest-NodeB-5373] *** the lock on NodeD is now stale, it won't be released by any commit/rollback command 2012-07-04 12:27:52,580 ERROR (OOB-7,ISPN,DistL1WriteSkewTest-NodeD-6770:___defaultcache) [org.infinispan.interceptors.InvocationContextInter ceptor] ISPN000136: Execution error org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [counter] for requestor [GlobalTransaction::2856:remote]! Lock held by [GlobalTransaction::1568:remote] at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:214) at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:187) at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:177) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:210) at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:137) at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:241)