org.jboss.resource.connectionmanager.BasePool.emptySubPool() will shut down the per-subject or per-criteria subpool if it is idle, following the change in JBAS-5929 to reduce memory usage.

It determines if the sub-pool is idle by using isEmpty(), however that only checks whether there are connections in the pool and does not look at checked out connections. So if all the connections from a pool are checked out, it will be shut down and the in-use connections destroyed when they are returned.

This can manifest as a problem if a connection if during a transaction a 1PC resource is used twice, and the pool is shutdown before the second use because it has no connection that are not in use but have not reached the idle time.

I haven't managed to produce a simple test case for this yet, however a complex application which triggers it right after the IdleRemover runs works correctly with any of these changes:
1) On AS 5.0.0 (which doesn't have JBAS-5929) instead of 5.1.0
2) Using datasource settings which mean it uses OnePool (which overrides emptySubPool to do nothing) instead of PoolBySubject

To trigger it will require:
1) PoolBySubject, PoolByCri or PoolBySubjectAndCri to be used
2) an idle timeout set
3) a 1 phase resource
4) a transaction which uses that 1PC resource twice
5) the idle timeout to run after the first getConnection() call but before the second.

I believe that it should check whether there are any connections checked out and not shutdown the pool in that case since it is not idle.