Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2329

unresponsive broker under high load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • AMQ 7.4.0.CR2
    • AMQ 7.3.0.GA
    • None
    • None
    • -
    • ARTEMIS-2317

      Incorporate upstream changes, including ARTEMIS-2214, to avoid the following situation:

      In a cluster with 5 master/slave pairs, one live broker becomes unresponsive. The backup broker becomes live, but the unresponsive broker remains unresponsive (not shut down).

      329 blocked threads, all waiting on:
      
      "Thread-1504284 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@87a85e1)" Id=1571781 BLOCKED on org.apache.activemq.artemis.utils.SoftValueHashMap@61d63c78 owned by "Thread-1504889 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@87a85e1)" Id=1572534
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:136)
              -  blocked on org.apache.activemq.artemis.utils.SoftValueHashMap@61d63c78
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.getMessage(PageCursorProviderImpl.java:115)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl.queryMessage(PageSubscriptionImpl.java:574)
              at org.apache.activemq.artemis.core.paging.cursor.PagedReferenceImpl.getPagedMessage(PagedReferenceImpl.java:101)
              -  locked org.apache.activemq.artemis.core.paging.cursor.PagedReferenceImpl@31fa3436
              at org.apache.activemq.artemis.core.paging.cursor.PagedReferenceImpl.getMessage(PagedReferenceImpl.java:88)
              at org.apache.activemq.artemis.core.server.impl.QueuePendingMessageMetrics.decrementMetrics(QueuePendingMessageMetrics.java:71)
              at org.apache.activemq.artemis.core.server.impl.QueueImpl.decDelivering(QueueImpl.java:3404)
              at org.apache.activemq.artemis.core.server.impl.QueueImpl.postAcknowledge(QueueImpl.java:3021)
              at org.apache.activemq.artemis.core.server.impl.RefsOperation.afterCommit(RefsOperation.java:157)
              -  locked org.apache.activemq.artemis.core.server.impl.QueueImpl@61763b40
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:546)
              -  locked org.apache.activemq.artemis.core.transaction.impl.TransactionImpl@4e2cbb01
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:38)
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:304)
              at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:236)
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
              at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
              at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$7/888473870.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
              Number of locked synchronizers = 1
              - java.util.concurrent.ThreadPoolExecutor$Worker@15eeae7f
      
      Which is waiting on:
      
      "Thread-1504889 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@87a85e1)" Id=1572534 RUNNABLE
              at io.netty.buffer.WrappedByteBuf.getByte(WrappedByteBuf.java:224)
              at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.getByte(ChannelBufferWrapper.java:180)
              at org.apache.activemq.artemis.core.paging.impl.Page.read(Page.java:195)
              at org.apache.activemq.artemis.core.paging.impl.Page.readFromMapped(Page.java:177)
              at org.apache.activemq.artemis.core.paging.impl.Page.read(Page.java:128)
              -  locked org.apache.activemq.artemis.core.paging.impl.Page@64858c85
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.readPage(PageCursorProviderImpl.java:170)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:151)
              -  locked org.apache.activemq.artemis.utils.SoftValueHashMap@61d63c78
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.getNumberOfMessages(PageSubscriptionImpl.java:1094)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.isDone(PageSubscriptionImpl.java:1008)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.checkDone(PageSubscriptionImpl.java:1083)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorInfo.addACK(PageSubscriptionImpl.java:1063)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl.processACK(PageSubscriptionImpl.java:852)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl.access$1100(PageSubscriptionImpl.java:62)
              at org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$PageCursorTX.afterCommit(PageSubscriptionImpl.java:1136)
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:546)
              -  locked org.apache.activemq.artemis.core.transaction.impl.TransactionImpl@476181ad
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl.access$100(TransactionImpl.java:38)
              at org.apache.activemq.artemis.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:304)
              at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:236)
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
              at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
              at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
              at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$7/888473870.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      
              Number of locked synchronizers = 1
              - java.util.concurrent.ThreadPoolExecutor$Worker@6c50069d
      
      

              fnigro Francesco Nigro
              rhn-support-shiggs Stephen Higgs
              Mikhail Krutov Mikhail Krutov
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: