-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
None
-
None
When running the Qpid Interop Test's amqp_large_content_test against the amq6 broker (5.16.0-SNAPSHOT built from master), the broker eventually enters a state where every test fails. Once this state is reached, the broker must be restarted in order to regain normal operation.
The test sends 1MB and 10MB messages of the various AMQP varilable-length types (currently binary, list, map, string and symbol). In the case of list and map, there are 4 variations for each message size in which the total message size is subdivided into several parts. For list 1MB test, the list or map will be subdivided as follows:
- 1 x 1MB string
- 16 x 64kB strings
- 256 x 4kB strings
- 4096 x 256B strings
but the total payload will be approx. the same for each case.
A typical test runs as follows:
python -m qpid_interop_test.amqp_large_content_test WARNING: Rhea Javascript shims not found Test Broker: ActiveMQ v.5.16.0-SNAPSHOT on Java/1.8.0_161 test_binary_AmqpNetLite->AmqpNetLite (__main__.BinaryTestCase) ... ok test_binary_AmqpNetLite->ProtonCpp (__main__.BinaryTestCase) ... ok ... (approx. 40 tests pass) test_map_ProtonPython2->ProtonCpp (__main__.MapTestCase) ... ok test_map_ProtonPython2->ProtonPython2 (__main__.MapTestCase) ... ok test_map_ProtonPython2->ProtonPython3 (__main__.MapTestCase) ... ERROR test_map_ProtonPython3->AmqpNetLite (__main__.MapTestCase) ... ERROR ... (all the rest fail)
Although the test almost always fails during the map portion of the test, if this section of the test is run on its own, it always passes. So it seems to me that the broker is suffering some sort of cumulative effect with large messages. In addition, the log file shows the following error:
2018-03-09 10:34:07,993 | WARN | Transport Connection to: tcp://0:0:0:0:0:0:0:1:54634 failed: org.apache.activemq.transport.amqp.AmqpProtocolException: Could not process AMQP commands | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///0:0:0:0:0:0:0:1:54634@5672 2018-03-09 10:37:08,079 | WARN | Transport Connection to: tcp://0:0:0:0:0:0:0:1:54632 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///0:0:0:0:0:0:0:1:54632@5672 2018-03-09 10:37:21,004 | WARN | Transport Connection to: tcp://0:0:0:0:0:0:0:1:54642 failed: org.apache.activemq.transport.amqp.AmqpProtocolException: Could not process AMQP commands | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///0:0:0:0:0:0:0:1:54642@5672 2018-03-09 10:37:21,569 | ERROR | Error in thread 'ActiveMQ BrokerService[localhost] Task-13' | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[localhost] Task-13 java.lang.OutOfMemoryError: Java heap space 2018-03-09 10:37:21,569 | ERROR | Error in thread 'ActiveMQ BrokerService[localhost] Task-11' | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[localhost] Task-11 java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)[:1.8.0_161] ... (see attached log file for full trace)
Looking at the AMQP traffic between the broker and clients, once this condition occurs, all attempts to use the AMQP transport result in the following error:
Condition: amqp:decode-error Description: Could not process AMQP commands
