XTS recovery tests failing due to a non-empty TXObjectStore after test complete

See: http://172.17.131.2/view/Narayana+BlackTie/job/narayana-java7/411/console

Looking at http://172.17.131.2/job/narayana-java7/411/artifact/XTS/localjunit/crash-recovery-tests/target/surefire-reports/com.arjuna.qa.junit.TestBASubordinateCrashDuringCommitAfterSubordinateExit.txt you will see:

java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at com.arjuna.qa.junit.BaseCrashTest.tearDown(BaseCrashTest.java:105)

This assertion checks that the TXObjectStore is empty after the test completes. For the three tests that failed in this job, the log was not empty.

Looking at the end of http://172.17.131.2/job/narayana-java7/411/artifact/XTS/localjunit/crash-recovery-tests/target/surefire-reports/com.arjuna.qa.junit.TestATHeuristicRecoveryAfterDelayedCommit-output.txt shows you:

�[0m�[0m10:36:04,605 INFO  [stdout] (Periodic Recovery) rule.debug{trace remove committed state} : removed committed transaction 0:ffffac118303:4e49dd0f:50110ed5:10
�[0m�[0m10:36:04,605 INFO  [stdout] (Periodic Recovery) rule.debug{trace remove committed state} : !!!killing JVM!!!

This suggests that the transaction log was removed. However, I'm not sure that the Byteman script checks that the participant record has also been removed. It's possible that the AS is being killed slightly too early, before the participant record is removed. We should check other scripts to see if they wait for both logs to be removed.

JBTM-1208 should help debug this problem if we get it again.