Copied description from BZ1255494 :

Description of problem:
It looks like the ejb3 subsystem thread pool configuration is hard coded to create an unbounded thread pool, where it it looks like max-threads = core-threads, and thus the threads will increase up to the max-threads configured and then remain there. The keep-alive setting which appears in many of the docs & default configurations is ineffective since max=core.

ejb3/src/main/java/org/jboss/as/ejb3/subsystem/EJB3SubsystemRootResourceDefinition.java

I tried defining a different thread pool in the threads subsystem and tried to reference it from the ejb3 subsystem, however it looks like the ejb3 subsystem only looks for thread pools configured in the ejb3 subsystem.