In conditions that are currently uncertain, but involve a large number of concurrent client connections, a Fuse application deadlocks whilst waiting for a free database connection out of a DBCP connection pool.

The thread dump shows hundreds of threads all exactly the same:

"qtp2016300324-1436" #1436 prio=5 os_prio=0 tid=0x00007f29a8736800 nid=0x1135 in Object.wait() [0x00007f28be0b8000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1118)
        - locked <0x00000007253e3b80> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)

 

There are no threads actually communicating with the database. In fact, there are no threads doing anything at all, except waiting for a free connection in the pool.

It would be easy to explain this on the basis that application components have been opening connections from the pool, and not returning them. However, in such a situation I should expect to see lots of active database connections. In fact, 'netstat' shows no more than four connections at any time – and that's four more than appear in the thread dump.