Uploaded image for project: 'A-MQ Broker'
  1. A-MQ Broker
  2. ENTMQBR-3864

[LTS] Potential deadlock when destroying a queue and depaging concurrently

    Details

      Description

      I've opened this JIRA since a customer reports  an issue described by ARTEMIS-2592 that was supposed to be fixed in AMQ 7.6. But they claim to be able to see the same issue with AMQ 7.7 HF 1.

      The broker JVM gets killed

      2020-08-10 15:21:06,210 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component QueueImpl[name=nonDurable.IM_lmprdrds010_baileym_2020-08-10T14:54:32:660.order_history, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=e98abd58-6b0a-11ea-9b95-7cd30ad7cc58], temp=true]@5a41e1a1 is not responsive 
      

      And after that broker log file shows deadlock that seems to be almost identical to ARTEMIS-2592.

      Deadlock detected!
      
      "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501 BLOCKED on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1 owned by "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3078)
          -  blocked on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2600(QueueImpl.java:126)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl$DepageRunner.run(QueueImpl.java:4079)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 2
          - java.util.concurrent.ThreadPoolExecutor$Worker@5798e685
          - java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
      
      
      "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643 owned by "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501
          at sun.misc.Unsafe.park(Native Method)
          -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:2073)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:2040)
          at org.apache.activemq.artemis.core.server.Queue.deleteMatchingReferences(Queue.java:296)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2030)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2025)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2228)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2213)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2161)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2152)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2132)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$201/1675316695.run(Unknown Source)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1425)
          -  locked org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:633)
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:566)
          -  locked org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@115ee963
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:540)
          at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1707)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:597)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:292)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$107/1704846988.onMessage(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@6123280d
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  jbertram Justin Bertram
                  Reporter:
                  dbruscin Domenico Francesco Bruscino
                  Tester:
                  Tiago Bueno
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: