Heap dump diagnostics revealed a memory leak in org.jboss.util.deadlock.DeadlockDetector.
The more deadlocks are detected the more "waiting" HashMap is growing.

In our case the HashMap is >311MB in size. It contains 20 entries, 15 of them are > 20MB in size.
The large size is not caused by map values but rather map keys which are of type com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple and contain a large number (couple thousand) of org.jboss.ejb.EntityEnterpriseContext references in its _txLocalResources variable.