Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-9092

AMQ 7.12 CR5: heap exhaustion with amqp-connection mirror down, even though addresses are paging

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • None
    • False
    • None
    • False
    • ARTEMIS-4798
    • Important

      This appears to be a new problem in 7.12 CR5; I could not reproduce it in CR3, although there were other problems related to mirroring and paging in CR3.

      The key to reproducing this problem is to produce to, and consume from, an address in such a way that there is a high throughput, but no backlog on the producer's address. That is, messages should be consumed as quickly as they are produced. Although the backlog on this address is near zero, there remains a backlog on the mirror's store-and-forward queue, because the messages cannot be forwarded.

      We expect that paging will take care of the memory used by this store-and-forward backlog. We set a value of maxSizeBytes that is smaller than globalMaxSize, and much smaller than -Xmx. However,  in CR5 paging seems not to help. Heap use continues to increase, and the addresses should memory usage in excess of their maxSizeBytes setting.

      It isn't clear to me at this stage whether paging isn't freeing the messages from memory, or something is not paging that should be.

      Steps to reproduce

      ---------------------------

      • Install AMQ 7.12 CR5
      • In the configuration, define an <amqp-connection> with a dummy target broker
      • Set maxSizeBytes=100Mb on all addresses
      • Establish a durable consumer on some address, that does no work
      • Establish a producer on the same address, than produces messages at full speed
      • Monitor the broker status and JVM heap usage

      Note that the logs will indicate that the application address has started paging. I'm unsure whether the mirror's store-and-forward address starts paging. However, memory used by both addresses will continue to increase beyond maxSizeBytes. Eventually the broker will run out of memory, although this might take many minutes.

       

       

              csuconic@redhat.com Clebert Suconic
              rhn-support-kboone Kevin Boone
              Messaging QE Messaging QE
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: