Non-JPA applications that use Hibernate with JTA must set hibernate.transaction.coordinator_class

By default, Hibernate will assume that a non-JPA application uses JDBC, which will cause problems if the application actually uses JTA-based transactions.

A non-JPA application that uses JTA-based transactions should explicitly set hibernate.transaction.coordinator_class=jta or provide a custom org.hibernate.resource.transaction.TransactionCoordinatorBuilder that builds a org.hibernate.resource.transaction.TransactionCoordinator that properly coordinates with JTA-based transactions.

See the migration guide for Hibernate ORM 5.0.x section about transactions for details:
https://github.com/hibernate/hibernate-orm/blob/5.0/migration-guide.adoc#transactions

The user guide will be updated with this information in 5.0.11. Those updates can be seen at:
https://github.com/hibernate/hibernate-orm/commit/f23b550b623a484dd56ccd1028fdaee34c64f907