Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-605

Streaming huge messages between cluster nodes causes java.lang.OutOfMemoryError: Direct buffer memory

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • AMQ 7.0.1.GA
    • A-MQ 7.0.0.ER16
    • None
    • None
    • Hide

      Steps to reproduce the issue 

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
scp jbossqa@10.40.4.81:~/tmp/jboss-eap-7.1.0.DR6-artemis-140.zip . # see private comment below for password
# change path per location of jboss-eap-7.1.0.DR6-artemis-140.zip
groovy -DEAP_ZIP_URL=file:///home/mnovak/tmp/jboss-eap-7.1.0.DR6-artemis-140.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
cd ../jboss-hornetq-testsuite/
mvn clean test -Dtest=HugeMessageTestCase#test1GbMessageCluster -Deap=7x   | tee log
      Show
      Steps to reproduce the issue git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ scp jbossqa@10.40.4.81:~/tmp/jboss-eap-7.1.0.DR6-artemis-140.zip . # see private comment below for password # change path per location of jboss-eap-7.1.0.DR6-artemis-140.zip groovy -DEAP_ZIP_URL=file: ///home/mnovak/tmp/jboss-eap-7.1.0.DR6-artemis-140.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=HugeMessageTestCase#test1GbMessageCluster -Deap=7x | tee log
    • AMQ Sprint 1

    Description

      Scenario

      • 2 servers are started in cluster.
      • Send one 1GB message to node1. Message is sent as described in subchapter Streaming over JMS in Artemis docs , it is sent like stream from file : 
        BytesMessage message = session.createBytesMessage();
message.setObjectProperty("JMS_AMQ_InputStream", bufferedInput);
producer.send(message);
      • Connect consumer on node2 and wait for message to redistribute to node2.
      • Receive 1GB stream message from node2 as described in subchapter Streaming over JMS in Artemis docs , it is saved to file. 
         BytesMessage messageReceived = (BytesMessage) consumer.receive(timeout);
// This will block until the entire content is saved on disk
messageReceived.setObjectProperty("JMS_AMQ_SaveStream", bufferedOutput);

      What is wrong
      During message redistribution from node1 to node2 following error appears. It is thrown on node1.

      OOM on node1 during message redistribution
      ERROR [org.apache.activemq.artemis.core.client] (Thread-9 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@1fb37a57-1791923158)) AMQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.8.0_65]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.8.0_65]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_65]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:673) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:234) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:218) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:364) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:268) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:249) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:834) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:842) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.writeBytes(ChannelBufferWrapper.java:554) [artemis-commons-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionContinuationMessage.encodeRest(SessionContinuationMessage.java:68) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionSendContinuationMessage.encodeRest(SessionSendContinuationMessage.java:98) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:285) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:251) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:220) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendServerLargeMessageChunk(ActiveMQSessionContext.java:460) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSendServer(ClientProducerImpl.java:388) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSend(ClientProducerImpl.java:330) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:266) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl$2.run(BridgeImpl.java:685) [artemis-server-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]

      Notes

      • This scenario only with one server (no cluster, just send/receive on one node) passes OK.
      • Message size is larger than server JVM memory. Artemis docs says following : "The only realistic limit to the size of a message that can be sent or consumed is the amount of disk space you have available".

      Customer impact: If large message (~GB) will be sent to server and redistributed/load-balanced to another server in cluster then source server will crash on OutOfMemoryError leaving any service unavailable.

      There is no known workaround.

      Update with 7.1.0.DR12

      The very same scenario now produces following error during message redistribution between cluster nodes. This is thrown on sending node.

      17:52:31,171 WARN  [org.apache.activemq.artemis.utils.OrderedExecutorFactory] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@58d509f4-431458715)) failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488): io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:624) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:578) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:686) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:675) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:372) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:120) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:284) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:265) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1036) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1044) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.writeBytes(ChannelBufferWrapper.java:570) [artemis-commons-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionContinuationMessage.encodeRest(SessionContinuationMessage.java:67) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionSendContinuationMessage.encodeRest(SessionSendContinuationMessage.java:96) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:291) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:277) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:273) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:225) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendServerLargeMessageChunk(ActiveMQSessionContext.java:468) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSendServer(ClientProducerImpl.java:379) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSend(ClientProducerImpl.java:323) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:261) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:126) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl$2.run(BridgeImpl.java:668) [artemis-server-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-6606 Streaming huge messages between cluster nodes causes java.lang.OutOfMemoryError: Direct buffer memory

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-695 Streaming huge messages between cluster nodes. Message received doesn't match message sent.

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          cloned to

          ARTEMIS-1098 Loading...

          Activity

            People

              fnigro Francesco Nigro
              mtaylor1@redhat.com Martyn Taylor (Inactive)
              Dmitrii Puzikov Dmitrii Puzikov (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: