Deadlock in StatefulSessionFilePersistenceManager when too many ejb clients for ejb3 cache maxSize

The problem is in SimpleStatefulCache.

Consider what happens when you load a passivated SFSB instance:
1) org.jboss.ejb3.cache.simple.SimpleStatefulCache.get() is called
2) it synchronizes on this.cacheMap and checks if it is in memory
3) when it's not, it enters the second part of the code, synchronizing on this.pm and then this.cacheMap
4) it checks for the passivated bean

So the locking order is the StatefulSessionPersistenceManager first and then the CacheMap second

Consider what happens when you create a new SFSB instance, which causes an old one to be passivated:
1) org.jboss.ejb3.cache.simple.SimpleStatefulCache.create() is called
2) after creating the instance, it synchronizes on this.cacheMap and calls cacheMap.put()
3) when the cache is full, CacheMap.removeEldestEntry() is called
4) if it can passivate the bean, it calls SimpleStatefulCache.passivate()
5) that method synchronizes on this.pm

So the locking order is CacheMap first and StatefulSessionPersistenceManager second.

Not having the same locking order will cause this deadlock if more than one thread is involved, and SimpleStatefulCache.get() and SimpleStatefulCache.create() are invoked concurrently.

Note: Altering the synchronization order in SimpleStatefulCache.get() will lead to a regression in https://issues.jboss.org/browse/EJBTHREE-1549.