Database connection is closed before the transaction handler is able to initiate an rollback on this connection.
Please take a look at forum entry: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=137613

Inside WrappedPreparedStatement method setDouble the Exception from underlying jdbc-driver is thrown. In case of this example
with Double.NaN as parameter value, Oracle causes an IllegalArgumentException instead of an SQLException.
Because of that BaseWrapperManagedConnection method connectionError is called:

Code:
void connectionError(Throwable t)

{ if (t instanceof SQLException == false || mcf.isExceptionFatal((SQLException) t)) broadcastConnectionError(t); }

This Throwable isn't an instance of SQLException so it seems to be an
connection error and broadcastConnectionError is called which causes
unregisterConnections and so the connection on jdbc level is closed
without rollback. Because of that the insert1 statement is commited
because oracle jdbc commits on close. So the transaction handling is not correct.