Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3812

Potential deadlock when destroying a queue and depaging concurrently

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: AMQ 7.7.0.GA
    • Fix Version/s: AMQ 7.8.0.CR1
    • Component/s: broker-core
    • Labels:
      None
    • Target Release:
    • Release Notes Text:
      Previously, if a queue was destroyed (for example when a non-durable consumer closed its connection) and depaging was concurrently happening, this might result in a deadlock condition. This issue is now resolved.
    • Release Notes Docs Status:
      Documented as Resolved Issue
    • Verified:
      Verified in a release

      Description

      I've opened this JIRA since a customer reports  an issue described by ARTEMIS-2592 that was supposed to be fixed in AMQ 7.6. But they claim to be able to see the same issue with AMQ 7.7 HF 1.

      The broker JVM gets killed

      2020-08-10 15:21:06,210 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component QueueImpl[name=nonDurable.IM_lmprdrds010_baileym_2020-08-10T14:54:32:660.order_history, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=e98abd58-6b0a-11ea-9b95-7cd30ad7cc58], temp=true]@5a41e1a1 is not responsive 
      

      And after that broker log file shows deadlock that seems to be almost identical to ARTEMIS-2592.

      Deadlock detected!
      
      "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501 BLOCKED on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1 owned by "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3078)
          -  blocked on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2600(QueueImpl.java:126)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl$DepageRunner.run(QueueImpl.java:4079)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 2
          - java.util.concurrent.ThreadPoolExecutor$Worker@5798e685
          - java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
      
      
      "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643 owned by "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501
          at sun.misc.Unsafe.park(Native Method)
          -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:2073)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:2040)
          at org.apache.activemq.artemis.core.server.Queue.deleteMatchingReferences(Queue.java:296)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2030)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2025)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2228)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2213)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2161)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2152)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2132)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$201/1675316695.run(Unknown Source)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1425)
          -  locked org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:633)
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:566)
          -  locked org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@115ee963
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:540)
          at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1707)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:597)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:292)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$107/1704846988.onMessage(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@6123280d
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbertram Justin Bertram
              Reporter:
              raggz Tom Ross
              Tester:
              Oleg Sushchenko Oleg Sushchenko
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: