1400+ TxCleanupService,___defaultcache,local and Scheduled-eviction-thread

In our application we are seeing

a. 700+threads with name TxCleanupService,___defaultcache,local

b. 700+ threads with name Scheduled-eviction-thread-*

They all have identical stacktrace and they are in TIMED_WAITING state basically not doing anything. It's making our application go unresponsive.

Here are few pointers:

• Complete thread dump of this application can be found here.
• This application is running with following version of Inifinispan library:

1. infinispan-core-6.0.2.Final.jar
2. infinispan-cachestore-jdbc-6.0.2.Final.jar
3. infinispan-client-hotrod-6.0.2.Final.jar
4. infinispan-commons-6.0.2.Final.jar
5. infinispan-cachestore-remote-6.0.2.Final
6. infinispan-tree-6.0.2.Final.jar
7. infinispan-cachestore-jdbc-6.0.2.Final.jar

Here are several other libraries present with infinispan in their filename like:

1. hibernate-infinispan-4.3.7.Final.jar
2. picketbox-infinispan-4.0.21.Final.jar
3. wildfly-clustering-infinispan-8.2.1.Final.jar
4. wildfly-clustering-ejb-infinispan-8.2.1.Final.jar
5. wildfly-clustering-web-infinispan-8.2.1.Final.jar

• Stack trace of TxCleanupService,___defaultcache,local & Scheduled-eviction-thread-* are given below

StackTrace of the threads

TxCleanupService,___defaultcache,local

java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000004ddcd7d98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  Locked ownable synchronizers:
• None

Scheduled-eviction-thread

java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000004ddd16ae8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at java.lang.Thread.run(Thread.java:748)
  Locked ownable synchronizers:
• None