A thread evicting key k may block to write an expiration record to expiryEntryQueue at the same time the expiration purge thread is trying to purge the same key k, causing a deadlock:
"remote-thread--p2-t23" #192 daemon prio=5 os_prio=0 tid=0x00007f160c00a000 nid=0xdc69 waiting on condition [0x00007f07f626d000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00007f0e4f0c9288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350) at org.infinispan.persistence.leveldb.LevelDBStore.addNewExpiry(LevelDBStore.java:600) at org.infinispan.persistence.leveldb.LevelDBStore.write(LevelDBStore.java:443) at org.infinispan.persistence.manager.PersistenceManagerImpl.writeToAllStores(PersistenceManagerImpl.java:500) at org.infinispan.persistence.manager.PersistenceManagerImpl.writeToAllStores(PersistenceManagerImpl.java:490) at org.infinispan.eviction.impl.PassivationManagerImpl.passivate(PassivationManagerImpl.java:87) at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryChosenForEviction(DefaultDataContainer.java:369) at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.notifyListenerOfRemoval(BoundedEquivalentConcurrentHashMapV8.java:2702) at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.replaceNode(BoundedEquivalentConcurrentHashMapV8.java:2760) - locked <0x00007f0edbcb4198> (a org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$Node) at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LRUEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:716) at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3682) at org.infinispan.container.DefaultDataContainer.put(DefaultDataContainer.java:235) at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:149) at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:100) at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:652) "pool-8-thread-1" #112 prio=5 os_prio=0 tid=0x00007f16800eb000 nid=0xdc18 waiting for monitor entry [0x00007f0a6e719000] java.lang.Thread.State: BLOCKED (on object monitor) at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3592) - waiting to lock <0x00007f0edbcb4198> (a org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$Node) at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:335) at org.infinispan.expiration.impl.ExpirationManagerImpl.handleInMemoryExpiration(ExpirationManagerImpl.java:135) at org.infinispan.expiration.impl.ClusterExpirationManager.processExpiration(ClusterExpirationManager.java:82) at org.infinispan.expiration.impl.ExpirationManagerImpl$ScheduledTask.run(ExpirationManagerImpl.java:232) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
The deadlock was fixed when switching to Caffeine in Infinispan 9.0 (ISPN-6998).