Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-3029

IllegalMonitorStateException in LockSupportCacheStore.loadAllKeys

    XMLWordPrintable

Details

    Description

      Most LockSupportCacheStore methods that call acquireGlobalLock() ignore its return value and proceed as if the lock was acquired on all the buckets. ISPN-2378 partially fixed this by only attempting to unlock the global lock if the lock was actually acquired, but the processing that was supposed to be protected by the global lock is still executed even if the lock acquisition failed.

      In loadAllKeys, this doesn't usually cause any problems. But if the cache store contains expired entries, it will try to upgrade a bucket lock to a write lock in order to update the bucket on disk, and the upgrade will fail with a IllegalMonitorStateException:

      > 20:41:36,960 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (undefined) Failed to execute outbound transfer: java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
      >       at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:447) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:431) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1340) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:883) [rt.jar:1.7.0_09-icedtea]
      >       at org.infinispan.util.concurrent.locks.StripedLock.upgradeLock(StripedLock.java:140) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.LockSupportCacheStore.upgradeLock(LockSupportCacheStore.java:106) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.bucket.BucketBasedCacheStore.access$000(BucketBasedCacheStore.java:49) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.bucket.BucketBasedCacheStore$CollectionGeneratingBucketHandler.handle(BucketBasedCacheStore.java:159) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.file.FileCacheStore.loopOverBuckets(FileCacheStore.java:102) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.bucket.BucketBasedCacheStore.loadAllKeysLockSafe(BucketBasedCacheStore.java:219) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.LockSupportCacheStore.loadAllKeys(LockSupportCacheStore.java:179) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.decorators.AbstractDelegatingStore.loadAllKeys(AbstractDelegatingStore.java:140) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.decorators.AsyncStore.loadKeys(AsyncStore.java:184) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.loaders.decorators.AsyncStore.loadAllKeys(AsyncStore.java:205) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:163) [infinispan-core-5.2.4.Final-redhat-2.jar:5.2.4.Final-redhat-2]
      >       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_09-icedtea]
      >       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_09-icedtea]
      >       at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_09-icedtea]
      >       at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.0.0.GA-redhat-2.jar:2.0.0.GA-redhat-2]
      

      A simple solution would be to throw an exception any time the global lock acquisition failed, but the current global lock acquisition algorithm might need to change because it seems very deadlock-prone at the moment.

      Attachments

        Issue Links

          Activity

            People

              pruivo@redhat.com Pedro Ruivo
              dberinde@redhat.com Dan Berindei
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: