An XAException with errorcode -7 is RMFAIL, which means that the transaction coordinator does not know what state the resource manager is in. It may have committed or it may have rolled back (it could have done this autonomously). This may be a permanent failure of the resource manager or it may only be transient; the transaction manager has no way of knowing. At this point the transaction outcome is potentially non-atomic (an heuristic outcome) and outside the scope of the TM to try to resolve automatically.
In ATS 4.0.2, the failure during commit was masked and commit would return committed. What happened was that the failed resource was retained in the transaction log and recovery would try to complete the transaction later. This silent behaviour was wrong (a bug): the application needs to know that a potential heuristic outcome has occurred. Therefore, in ATS 4.0.3 the error is handled differently (correctly): the transaction will throw a heuristic exception and, as before, retain the reference to the XAResource in the hopes that the log can be used to resolve the potential heuristic.
The failure recovery subsystem retry mechanism has not changed. However, because the failure reported from the XAResource has changed, this impacts what failure recovery will do. In 4.0.2, the error (wrongly) was reported as being something that failure recovery could automatically try to deal with. Hence, when the recovery manager ran and found the log entry that referenced your XAResource, it would try to complete the transaction. In 4.0.3, the error is a heuristic, which cannot be managed automatically: that is the whole point of a heuristic error. It needs manual intervention. In this case, the recovery manager will ignore the log entry and do nothing.