Lets have a look at JDBCCacheLoader.java:243. Method remove(Fqn fqn). It acquires CONNECTION and then tries to acquire LOCK.
Lets have a look at JDBCCacheLoader.java:138. Method put(Fqn name, Map attributes). It acquires LOCK and then calls addNewSubtree(..) or updateNode(..) and they are getting CONNECTION.
Concurrent removes and puts to/from cache with JDBC cache loader, having fixed pool of database connections.
One thread invokes JDBCCacheLoader.remove(Fqn fqn) and:
1. Acquires CONNECTION to DB from pool.
2. Tries to LOCK fqn, but it is locked by another thread.
now Connection is taken from pool, but is not currently used, cause waiting a lock.
Another thread invokes JDBCCacheLoader.put(Fqn name, Map attributes) and:
1. Acquires LOCK on fqn
2. invokes addNewSubtree(..) or updateNode(..), and one of those methods tries to get CONNECTION from pool. But pool exhausted and deadlock appears.
Imagine that we have pool size = 1. So classic deadlock takes place (L-lock resource, C - connection resource):
gets C, L
gets L, C.
(1st acquires C and 2nd acquires L, then 1st waits for L and second waits for C)
When we have pool size 50..100 and many (900) threads such situation appears very often.
Must admit, that AdjListJDBCCacheLoader.getChildrenNames(Fqn fqn) (line : 190), also opens connection and then tries to acquire lock on fqn.
Possible way to solve is to acquire lock, before opening the connection to DB in methods:
AdjListJDBCCacheLoader.getChildrenNames(...) (line : 190)
JDBCCacheLoader.remove(...) (line : 254)