I encountered this unexpected situation while running an application in JBoss EAP 8.

In a @Stateless EJB, a (transactional) method is calling another @Stateless EJB (transactional) method, which is failing. The transaction propagation is the default one, REQUIRED, for both. Here's the skeleton:

public MyObj outer() throws MyApplicationException {
  try { 
    innerEjb.call(); 
  } finally { 
    myEntity = entityManager.merge(myEntity); 
  }
}

The call to innerEjb.call() is failing (that method is throwing a MyApplicationException, which is marked with @ApplicationException and rollback=true), so the current transaction is marked as rollbackOnly. So far so good. However, the call to entityManager.merge is failing with a jakarta.persistence.TransactionRequiredException. However, this is unexpected, because a transaction is actually in place, even if marked as rollbackOnly. So, even if I expect the final call to outer to fail, and the running transaction to be rolled back, I don't expect to get the TransactionRequiredException, but rather my own MyApplicationException as thrown by the innerEjb.call() call, as the final result to the outer() call.

By debugging, I found out that the origin of the problem lies in:
org.jboss.as.jpa.transaction.TransactionUtil.isInTx(TransactionManager)
which calls org.jboss.tm.TxUtils.isActive(Transaction) which in turn returns true only if the transaction is active (status == 4), but not if it's rollbackOnly (status == 1). This causes the TransactionRequiredException exception to be thrown in the end, by org.jboss.as.jpa.container.AbstractEntityManager.transactionIsRequired().
By looking at the source of TxUtils, it seems like to me that the correct method to be called would be isUncommitted(Transaction) instead, which indeed returns true if the status is either active or rollbackOnly.
Indeed, if I search on the Internet I find an older version of this TxUtils class for which the isActive method was returning true exactly in those two cases:
https://docs.jboss.org/jbossas/javadoc/4.0.2/org/jboss/tm/TxUtils.java.html
So, maybe this is the origin of the problem, unless the change in behavior was instead done on purpose, but then I'd like to know why the current behavior is the expected one, because I find it inconsistent.