Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3864

[LTS] Potential deadlock when destroying a queue and depaging concurrently

XMLWordPrintable

      I've opened this JIRA since a customer reports  an issue described by ARTEMIS-2592 that was supposed to be fixed in AMQ 7.6. But they claim to be able to see the same issue with AMQ 7.7 HF 1.

      The broker JVM gets killed

      2020-08-10 15:21:06,210 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component QueueImpl[name=nonDurable.IM_lmprdrds010_baileym_2020-08-10T14:54:32:660.order_history, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=e98abd58-6b0a-11ea-9b95-7cd30ad7cc58], temp=true]@5a41e1a1 is not responsive 
      

      And after that broker log file shows deadlock that seems to be almost identical to ARTEMIS-2592.

      Deadlock detected!
      
      "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501 BLOCKED on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1 owned by "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.depage(QueueImpl.java:3078)
          -  blocked on org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$2600(QueueImpl.java:126)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl$DepageRunner.run(QueueImpl.java:4079)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 2
          - java.util.concurrent.ThreadPoolExecutor$Worker@5798e685
          - java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
      
      
      "Thread-25 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@7bd4937b)" Id=178 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643 owned by "Thread-1820 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$7@1de76cc7)" Id=5501
          at sun.misc.Unsafe.park(Native Method)
          -  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2029c643
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
          at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
          at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.iterQueue(QueueImpl.java:2073)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteMatchingReferences(QueueImpl.java:2040)
          at org.apache.activemq.artemis.core.server.Queue.deleteMatchingReferences(Queue.java:296)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2030)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:2025)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.deleteQueue(QueueImpl.java:2228)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2213)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2161)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2152)
          at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.destroyQueue(ActiveMQServerImpl.java:2132)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl.doIt(TransientQueueManagerImpl.java:43)
          at org.apache.activemq.artemis.core.server.impl.TransientQueueManagerImpl$$Lambda$201/1675316695.run(Unknown Source)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.execute(ReferenceCounterUtil.java:81)
          at org.apache.activemq.artemis.utils.ReferenceCounterUtil.decrement(ReferenceCounterUtil.java:71)
          at org.apache.activemq.artemis.core.server.impl.QueueImpl.removeConsumer(QueueImpl.java:1425)
          -  locked org.apache.activemq.artemis.core.server.impl.QueueImpl@5a41e1a1
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.removeItself(ServerConsumerImpl.java:633)
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:566)
          -  locked org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl@115ee963
          at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:540)
          at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:1707)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.slowPacketHandler(ServerSessionPacketHandler.java:597)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler.onMessagePacket(ServerSessionPacketHandler.java:292)
          at org.apache.activemq.artemis.core.protocol.core.ServerSessionPacketHandler$$Lambda$107/1704846988.onMessage(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.Actor.doTask(Actor.java:33)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
          at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
          at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$29/910599202.run(Unknown Source)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@6123280d
      

              rhn-support-jbertram Justin Bertram
              dbruscin Domenico Francesco Bruscino
              Tiago Bueno Tiago Bueno
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: