Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-2845

Deadlock in LevelDBStore with passivation enabled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JDG 7.2
    • JDG 7.1.2 GA
    • Core
    • None

    Description

      A thread evicting key k may block to write an expiration record to expiryEntryQueue at the same time the expiration purge thread is trying to purge the same key k, causing a deadlock:

      "remote-thread--p2-t23" #192 daemon prio=5 os_prio=0 tid=0x00007f160c00a000 nid=0xdc69 waiting on condition [0x00007f07f626d000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00007f0e4f0c9288> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350)
      	at org.infinispan.persistence.leveldb.LevelDBStore.addNewExpiry(LevelDBStore.java:600)
      	at org.infinispan.persistence.leveldb.LevelDBStore.write(LevelDBStore.java:443)
      	at org.infinispan.persistence.manager.PersistenceManagerImpl.writeToAllStores(PersistenceManagerImpl.java:500)
      	at org.infinispan.persistence.manager.PersistenceManagerImpl.writeToAllStores(PersistenceManagerImpl.java:490)
      	at org.infinispan.eviction.impl.PassivationManagerImpl.passivate(PassivationManagerImpl.java:87)
      	at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryChosenForEviction(DefaultDataContainer.java:369)
      	at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.notifyListenerOfRemoval(BoundedEquivalentConcurrentHashMapV8.java:2702)
      	at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.replaceNode(BoundedEquivalentConcurrentHashMapV8.java:2760)
      	- locked <0x00007f0edbcb4198> (a org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$Node)
      	at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LRUEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:716)
      	at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3682)
      	at org.infinispan.container.DefaultDataContainer.put(DefaultDataContainer.java:235)
      	at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:149)
      	at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:100)
      	at org.infinispan.interceptors.locking.ClusteringDependentLogic$DistributionLogic.commitSingleEntry(ClusteringDependentLogic.java:652)
      	
      "pool-8-thread-1" #112 prio=5 os_prio=0 tid=0x00007f16800eb000 nid=0xdc18 waiting for monitor entry [0x00007f0a6e719000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3592)
      	- waiting to lock <0x00007f0edbcb4198> (a org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$Node)
      	at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:335)
      	at org.infinispan.expiration.impl.ExpirationManagerImpl.handleInMemoryExpiration(ExpirationManagerImpl.java:135)
      	at org.infinispan.expiration.impl.ClusterExpirationManager.processExpiration(ClusterExpirationManager.java:82)
      	at org.infinispan.expiration.impl.ExpirationManagerImpl$ScheduledTask.run(ExpirationManagerImpl.java:232)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

      The deadlock was fixed when switching to Caffeine in Infinispan 9.0 (ISPN-6998).

      Attachments

        Activity

          People

            ttarrant@redhat.com Tristan Tarrant
            dberinde@redhat.com Dan Berindei (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: