Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1004

[AMQP, Broker] periodic excessive CPU usage and many instances of org.apache.activemq.transport.amqp.AmqpProtocolConverter$ConsumerContext created

XMLWordPrintable

    • Hide

      Need to manipulate the thread order to exercise the race condition so I needed to use a DEBUGGER.

      1) Set a (*thread only*) breakpoint at line number 268 - "onConnectionOpen();"

      if (protonConnection.getLocalState() == EndpointState.UNINITIALIZED && protonConnection.getRemoteState() != EndpointState.UNINITIALIZED) {
                          onConnectionOpen();
       }
      

      2) Run the attached unit test in DEBUG mode. The unit test creates a subscriber.

      3)The unit test triggers the breakpoint (from point 1). After a few seconds it tears down the connection between client and broker. In the console output is

      15-05-10 11:05:20,510 | INFO  | Thread-2 | About to close PROXY...
      2015-05-10 11:05:20,510 | INFO  | Thread-2 | close, numConnections=1
      2015-05-10 11:05:20,511 | INFO  | Thread-2 | PROXY closed.
      

      4) Release the breakpoint.

      5) The test remains running for 5 minutes. You can see the CPU usage for the process rise and see error message similar to following logged

      ~~~~
      2015-05-10 11:05:58,974 | WARN | ActiveMQ Transport: tcp:///127.0.0.1:54122@54114 | Transport Connection to: tcp://127.0.0.1:54122 failed: java.net.SocketException: Broken pipe
      Exception in thread "Activ
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp+nio+ssl://0.0.0.0:0"
      eMQ Transport Server: amqp+ssl://0.0.0.0:0" Exception in thread "ActiveMQ Transport Server Thread Handler: amqp+nio+ssl://0.0.0.0:0"
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server Thread Handler: amqp+ssl://0.0.0.0:0"

      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp://0.0.0.0:0"
      Exception in thread "Reference Handler"
      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp+ssl://0.0.0.0:0"
      java.lang.OutOfMemoryError: GC overhead limit exceeded

      Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference Handler"
      org.apache.activemq.transport.amqp.AmqpProtocolException: Could not process AMQP commands
      Exception in thread "ActiveMQ Transport Server Thread Handler: amqp+nio://0.0.0.0:0" java.lang.OutOfMemoryError: GC overhead limit exceeded
      Exception in thread "ActiveMQ Transport Server: amqp+nio://0.0.0.0:0" java.lang.OutOfMemoryError: Java heap space
      ~~~~

      Show
      Need to manipulate the thread order to exercise the race condition so I needed to use a DEBUGGER. 1) Set a (* thread only *) breakpoint at line number 268 - "onConnectionOpen();" if (protonConnection.getLocalState() == EndpointState.UNINITIALIZED && protonConnection.getRemoteState() != EndpointState.UNINITIALIZED) { onConnectionOpen(); } 2) Run the attached unit test in DEBUG mode. The unit test creates a subscriber. 3)The unit test triggers the breakpoint (from point 1). After a few seconds it tears down the connection between client and broker. In the console output is 15-05-10 11:05:20,510 | INFO | Thread -2 | About to close PROXY... 2015-05-10 11:05:20,510 | INFO | Thread -2 | close, numConnections=1 2015-05-10 11:05:20,511 | INFO | Thread -2 | PROXY closed. 4) Release the breakpoint. 5) The test remains running for 5 minutes. You can see the CPU usage for the process rise and see error message similar to following logged ~~~~ 2015-05-10 11:05:58,974 | WARN | ActiveMQ Transport: tcp:///127.0.0.1:54122@54114 | Transport Connection to: tcp://127.0.0.1:54122 failed: java.net.SocketException: Broken pipe Exception in thread "Activ Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp+nio+ssl://0.0.0.0:0" eMQ Transport Server: amqp+ssl://0.0.0.0:0" Exception in thread "ActiveMQ Transport Server Thread Handler: amqp+nio+ssl://0.0.0.0:0" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server Thread Handler: amqp+ssl://0.0.0.0:0" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp://0.0.0.0:0" Exception in thread "Reference Handler" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "ActiveMQ Transport Server: amqp+ssl://0.0.0.0:0" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference Handler" org.apache.activemq.transport.amqp.AmqpProtocolException: Could not process AMQP commands Exception in thread "ActiveMQ Transport Server Thread Handler: amqp+nio://0.0.0.0:0" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "ActiveMQ Transport Server: amqp+nio://0.0.0.0:0" java.lang.OutOfMemoryError: Java heap space ~~~~

      It seems there is a race condition between org.apache.activemq.transport.amqp.AmqpProtocolConverter and org.apache.activemq.broker.TransportConnection

      • has the potential to create a tight while loop that creates a huge number of org.apache.activemq.transport.amqp.AmqpProtocolConverter$ConsumerContext instances and leaves the A-MQ broker unresponsive.

      Note:

      • Tested with <qpid-jms-version>0.26</qpid-jms-version> and <qpid-jms-version>0.30</qpid-jms-version> - same in both.
      • exact position of breakpoint above is important, otherwise it can trigger other client side timeouts that do not exercise race condition

              Unassigned Unassigned
              rhn-support-pfox Patrick Fox (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: