The issue was spotted in 6 hr soak tests and affects both distributed & replicated mode.

The test shows steady increase in heap usage. After closer examination of jfr recording, it seems that at the end of the test there are almost 22M live org.infinispan.transaction.xa.GlobalTransaction instances in the heap, totaling 667 MB in size.

Top heap consumers (final stage of the test)
Class Instances Size(bytes) Percentage of Heap(%)
byte[] 60,218 830,445,604 35.168

Class Instances Size(bytes) Percentage of Heap(%)
org.infinispan.transaction.xa.GlobalTransaction 21,846,176 699,077,616 29.605

Class Instances Size(bytes) Percentage of Heap(%)
java.util.concurrent.ConcurrentHashMap$Node 21,795,718 697,462,992 29.537

Class Instances Size(bytes) Percentage of Heap(%)
java.util.concurrent.ConcurrentHashMap$Node[] 129 134,352,464 5.69

Transaction configuration:
<transaction transaction-manager-lookup="org.infinispan.transaction.lookup.GenericTransactionManagerLookup" mode="NON_DURABLE_XA" />

Please let me know if you'd like me to share more details from jfr recording.