Transaction boundary commands can be completed before state transfer applies an active transaction

There is a condition where a transaction boundary command, like a RollbackCommand, can be processed by the new owner of a tx-encapsulated key before it applies the active transaction.

On a view change:

The forwarding node waits until its transactions from the new topology ID is set before issuing the command.

2013-09-17 14:28:33,725 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Invoked with command RollbackCommand

{gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1}

and InvocationContext [org.infinispan.context.impl.LocalTxInvocationContext@1f225e51]
2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferInterceptor] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) handleTopologyAffectedCommand for command RollbackCommand

{gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=-1}

2013-09-17 14:28:33,727 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) Waiting for transaction data for topology 26, current topology is 25
...
2013-09-17 14:28:34,101 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (OOB-350,session-resource-cluster,east-dht5-60816(CMC-Denver-CO)) east-dht5-60816(CMC-Denver-CO) broadcasting call RollbackCommand

{gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:local, cacheName='eigAllocation', topologyId=26}

to recipient list [west-dht4-48045(CH2-Chicago-IL), east-dht5-60816(CMC-Denver-CO), east-dht2-60243(CMC-Denver-CO)]

However, the receiving node has not finished applying the transactions from the forwarding node yet for this topology ID. So the tx cannot be found, resulting in a stale tx and an unusable key.

2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Calling perform() on RollbackCommand

{gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26}

2013-09-17 14:28:34,116 TRACE [org.infinispan.commands.tx.AbstractTransactionBoundaryCommand] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Did not find a RemoteTransaction for GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote
2013-09-17 14:28:34,116 TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (OOB-153,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) About to send back response SuccessfulResponse

{responseValue=null}

for command RollbackCommand

{gtx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote, cacheName='eigAllocation', topologyId=26}

...
2013-09-17 14:28:34,161 TRACE [org.infinispan.transaction.TransactionTable] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Created and registered remote transaction RemoteTransaction{modifications=[PutKeyValueCommand

{key=EdgeResourceCacheKey[edgeDeviceId=4109,resourceId=16825], value=EdgeInputResource [edgeInputId=16825, currentBandwidth=0, maxBandwidth=1048576000], flags=null, putIfAbsent=false, lifespanMillis=-1, maxIdleTimeMillis=-1, successful=true, ignorePreviousValue=false}

], lookedUpEntries={}, lockedKeys=null, backupKeyLocks=null, missingLookedUpEntries=false, isMarkedForRollback=false, tx=GlobalTransaction:<east-dht5-60816(CMC-Denver-CO)>:48002:remote}
2013-09-17 14:28:34,733 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (OOB-408,session-resource-cluster,west-dht4-48045(CH2-Chicago-IL)) Signalling transaction data received for topology 26