Order in which JBoss nodes are restarted affects the outcome of XA transactions recovery

The test case consisting of a servlet and two stateless session beans.

The servlet and SLSB A are deployed on node A and SLSB B is deployed on node B.

The servlet calls SLSB A on node A. The bean then retrieves XA data source from JNDI, creates JDBC connection and inserts a row into a table in the database (Postgres).
Next it retrieves JmsXa connection factory from JNDI, creates connection and sends a message to a queue.
After that it calls SLSB B on node B that retrieves XA data source from JNDI, creates JDBC connection and inserts a row into a table in the database.

At the end it should produce two rows in the database table and a message on the queue.

Node A has a simple byteman rule that kills node A JVM at the end of BasicAction.prepare() method. After JVM on node A dies, the JVM on node B gets "kill -9".

The above test case produces two different results that depend on the order in which JBoss nodes are restarted.

If node is restarted A first followed by node B the outcome is consistent result i.e. two rows can be seen in the database.

If the node B is restarted first followed by node A the outcome is inconsistent i.e. only one row in the table; row inserted by SLSB A on node A.

Both beans use the default attributes for transaction support and both nodes have been configured to use JTS transactions.

During recovery node B reports the following error:

2010-11-22 10:25:37,916 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (Thread-22) PeriodicRecovery: background thread backing off
2010-11-22 10:25:37,922 WARN [com.arjuna.ats.jta.logging.loggerI18N] (RequestProcessor-2) [com.arjuna.ats.internal.jta.resources.jts.orbspecific.xaerror] [com.arjuna.ats.internal.jta.resources.jts.orbspecific.xaerror] XAResourceRecord.rollback caused an XA error: XAException.XAER_RMERR
2010-11-22 10:25:37,922 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] (RequestProcessor-2) OutputObjectState::OutputObjectState()

Where as node A reports

2010-11-22 10:28:15,174 WARN [com.arjuna.ats.jts.logging.loggerI18N] (Thread-22) [com.arjuna.ats.internal.jts.resources.rrcaught] ResourceRecord commit - caught exception: org.omg.CORBA.OBJECT_NOT_EXIST: Server-side Exception: unknown oid vmcid: 0x0 minor code: 0 completed: No

Postgres log contains the following error:

2010-11-22 10:25:37 GMTERROR: prepared transaction with identifier "131072_Mi0tM2Y1N2ZlOWM6YzQ5ZTo0Y2VhNDM4MDpmYQ==_LTNmNTdmZTkyOjRhZDY6NGNlYTQzOGQ6ZmQ=" does not exist
2010-11-22 10:25:37 GMTSTATEMENT: ROLLBACK PREPARED '131072_Mi0tM2Y1N2ZlOWM6YzQ5ZTo0Y2VhNDM4MDpmYQ==_LTNmNTdmZTkyOjRhZDY6NGNlYTQzOGQ6ZmQ='