Deadlock scenario:

1. Transaction started. At some point, jdbc connection allocated from poll and registered as transaction resource.
2. After timeout event occured, JBoss try to rollback this transaction.
3. In process of rolling back transaction, reaper process try to get lock on connection to do rollback and stopped at this point in waiting.
4. Due to cancelling state of the transaction, exception is rised in main process of transaction.
5. Spring catched this exception and invokes rollback to transaction.
6. Inside jboss transaction manager there is synchronized method for making transaction abort, so this rollback invocation is waiting another thread to complete.
7. At final: JBoss hold synchronized lock on abort method, but wait for releasing lock on connection. Main process hold lock on connection, but wait for releasing lock on abort method. Deadlock state.

How this can be fixed:

Before trying to invoke abort method, check transaction state. If it is already in state of aborting - just rise IllegalStateException.

I'm also suggest to add similar check in spring code (https://jira.springsource.org/browse/SPR-8421), but such check should be in jboss code too, because any client code can suddenly get into this deadlock state.