Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-8689

Encountered OOME with majority of the heap loaded in the amqp connection pending persistentSize

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Undefined Undefined
    • None
    • AMQ 7.11.4.GA
    • broker-core
    • False
    • None
    • False
    • Medium
    • ARTEMIS-4651

      7 GB memory VM 

      4GB max heap (and now with 6GB). 

      1GB global max size

      no clustering

      one amqp connection mirroring  

      Encountered OOME: 

      2024-01-08 22:04:05,517 ERROR [org.apache.activemq.artemis.core.journal.impl.JournalImpl] Exception during appendAddRecord:
      java.lang.OutOfMemoryError: Java heap space
      » at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap$Section.rehash(ConcurrentLongHashMap.java:452) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap$Section.put(ConcurrentLongHashMap.java:334) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap.put(ConcurrentLongHashMap.java:128) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.core.journal.impl.JournalImpl$1.run(JournalImpl.java:937) ~[artemis-journal-2.28.0.redhat-00012.jar:2.28.0.redhat-00012]
      » at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
      » at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$198/0x00000008003af220.run(Unknown Source) ~[?:?]
      » at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
      » at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
      » at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.redhat-00012.jar:?]

      Heap dump shows: 

       

      1. 

      One instance of "org.apache.activemq.artemis.core.server.impl.QueueImpl" loaded by "java.net.URLClassLoader @ 0x700000000" occupies 4,039,332,072 (94.66%) bytes.

       
      2. 
      OQL to gather what's in the impl (ool.png) (screenshot)

       

      name     | consumers.consumers.size | pendingMetrics.messageCount | a.pendingMetrics.persistentSize | a.deliveringMetrics.messageCount | a.deliveringMetrics.persistentSize
      ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      $ACTIVEMQ_ARTEMIS_MIRROR_-01-to-02 |       1 |   282,612 |  3,149,248,978 |  1,379 | 144,795
      Q0002                              |   1,114 |       679 |      7,626,740 |      0 |       0    
      DLQ                                |       0 |         1 |         11,327 |      0 |       0    
      DR_BROKER:61616?sslEnabled=true    |       0 |         6 |          7,182 |      0 |       0    
      -------------------------------------------------------------------------------------------------------------------------------------------------------------------------
       

              csuconic@redhat.com Clebert Suconic
              rhn-support-whui Roger Hui
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: