Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1180

Offline Durable Topic Subscription exceeds memory limits

XMLWordPrintable

    • Hide

      This can be reproduced using the attached broker config in JBoss Fuse 6.2 using the following test from apache-activemq-5.11.0.redhat-620133/examples/openwire/swissarmy examples:

      1) Start two durable subscriptions using the openwire swissarmy examples:

      ant consumer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -Dmax 500000

      ant consumer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -DclientId=c2

      2) Start a producer using the following command:

      ant producer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -Dmax=500000

      3) Kill one of the durable topic subscribers and let the test run until the OOM

      4) Take a heap dump and review the memory usage of the durable topic subscription

      Show
      This can be reproduced using the attached broker config in JBoss Fuse 6.2 using the following test from apache-activemq-5.11.0.redhat-620133/examples/openwire/swissarmy examples: 1) Start two durable subscriptions using the openwire swissarmy examples: ant consumer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -Dmax 500000 ant consumer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -DclientId=c2 2) Start a producer using the following command: ant producer -Dtopic=true -Ddurable=true -Dsubject=test -Duser=admin -Dpassword=admin -Dmax=500000 3) Kill one of the durable topic subscribers and let the test run until the OOM 4) Take a heap dump and review the memory usage of the durable topic subscription

      Using durable topic subscriptions it is possible for offline durable topic subscriptions to exceed the destination memory limit as well as the system memory limit.I have broker which is configured as follows:| <destinationPolicy>|

      <policyMap>
      <policyEntries>
      <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb" usePrefetchExtension="false">
      </policyEntry>
      <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
      </policyEntry>
      </policyEntries>
      </policyMap>
      </destinationPolicy>
       
      ...
       
      <systemUsage>
      <systemUsage>
      <memoryUsage>
      <memoryUsage limit="250 mb"/>
      </memoryUsage>
      <storeUsage>
      <storeUsage limit="100 gb"/>
      </storeUsage>
      <tempUsage>
      <tempUsage limit="50 gb"/>
      </tempUsage>
      </systemUsage>
      </systemUsage>

      Full config is attached.In this scenario there are two durable topic subscription, though one only consumes a few messages and then goes offline while the other remains online and consumes messages.The offline durable topic subscription will continue to receive pending messages until the heap is exhausted causing an "java.lang.OutOfMemoryError: GC overhead limit exceeded" to occur.in my test a heap dump reveal the following about the durable subscription memory usage:One instance of "org.apache.activemq.broker.region.DurableTopicSubscription" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x7e1a916d0" occupies 395,238,672 (76.24%) bytes. The memory is accumulated in one instance of "org.apache.activemq.broker.region.cursors.OrderedPendingList" loaded by "org.apache.felix.framework.BundleWiringImpl$BundleClassLoaderJava5 @ 0x7e1a916d0".Or:Class Name | Objects | Shallow Heap | Retained Heap
      ----------------------------------------------------------------------------------------------------
      org.apache.activemq.broker.region.DurableTopicSubscription| 2 | 304 | >= 395,240,480
      ----------------------------------------------------------------------------------------------------This shows that one instance on the Durable Topic Subscription is using 395MB when it should have been limited to 1MB. This also far exceeds the brokers system memory limit which was configured at 250MB.While the test is running you can observe that offline durable subscription's CursorMemoryUsage appears to jump up and down in values as it would be expected if messages were actually being enqueued and dequeued. But in this case the PendingQueueSize is constantly increasing.

              gtully@redhat.com Gary Tully
              rhn-support-jsherman Jason Sherman
              Dominik Lenosi Dominik Lenosi (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: