Data Inconsistency when Recovery + syncCommitPhase=false

with syncCommitPhase=false, the CommitCommand is sent asynchronously. the TransactionCoordinator sends immediately the TxCompletionNotificationCommand that can be deliver first than the CommitCommand. The CommitCommand fails silently:

if (transaction == null) {
  if (trace) log.tracef("Did not find a RemoteTransaction for %s", globalTx);
    return invalidRemoteTxReturnValue();
  }
}

This bug affects the 5.3 and 5.2.5. I've made one test case to catch this bug:

5.3 => https://github.com/pruivo/infinispan/blob/rec-async/core/src/test/java/org/infinispan/tx/recovery/RecoveryWithAsyncCommitPhase.java

5.2 => https://github.com/pruivo/infinispan/blob/rec-async-5.2/core/src/test/java/org/infinispan/tx/recovery/RecoveryWithAsyncCommitPhase.java

Note: this bug may happen if you use async communication (prepare in 1PC)
Note2: this may be related to https://issues.jboss.org/browse/ISPN-2719

Possible solutions:

• do not allow to configure the cache with syncCommitPhase=false && recovery enabled;
• force syncCommitPhase=true when recovery is enabled;
• send the CommitCommand and the TxCompletionNotificationCommand as Regular Messages (they will be deliver in FIFO order)

Thanks to Diego Didona that spotted this bug.