Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: 7.1.0.DR18
Affects Version/s: 7.1.0.DR6, 7.1.0.DR12
Component/s: ActiveMQ
Labels:
- eap7.1-risks-mitigation

CDW devel_ack:
CDW docs_ack:
CDW pm_ack:
CDW qa_ack:
CDW release:
Target Release:

7.1.0.GA
Steps to Reproduce:
Hide

Steps to reproduce the issue (update for DR17)

git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_VERSION=7.1.0.DR17 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=HugeMessageTestCase#test1GbMessageCluster -Deap=7x | tee log
Show
Steps to reproduce the issue (update for DR17) git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_VERSION=7.1.0.DR17 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=HugeMessageTestCase#test1GbMessageCluster -Deap=7x | tee log

Sprint:
AMQ Sprint 1

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Scenario

2 servers are started in cluster.

Send one 1GB message to node1. Message is sent as described in subchapter Streaming over JMS in Artemis docs , it is sent like stream from file :

BytesMessage message = session.createBytesMessage();
message.setObjectProperty("JMS_AMQ_InputStream", bufferedInput);
producer.send(message);

Connect consumer on node2 and wait for message to redistribute to node2.

Receive 1GB stream message from node2 as described in subchapter Streaming over JMS in Artemis docs , it is saved to file.

 BytesMessage messageReceived = (BytesMessage) consumer.receive(timeout);
// This will block until the entire content is saved on disk
messageReceived.setObjectProperty("JMS_AMQ_SaveStream", bufferedOutput);

What is wrong
During message redistribution from node1 to node2 following error appears. It is thrown on node1.

OOM on node1 during message redistribution

ERROR [org.apache.activemq.artemis.core.client] (Thread-9 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@1fb37a57-1791923158)) AMQ214017: Caught unexpected Throwable: java.lang.OutOfMemoryError: Direct buffer memory
	at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.8.0_65]
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.8.0_65]
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_65]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:673) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:234) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:218) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:364) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:118) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:268) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:249) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:834) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:842) [netty-all-4.0.35.Final-redhat-1.jar:4.0.35.Final-redhat-1]
	at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.writeBytes(ChannelBufferWrapper.java:554) [artemis-commons-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionContinuationMessage.encodeRest(SessionContinuationMessage.java:68) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionSendContinuationMessage.encodeRest(SessionSendContinuationMessage.java:98) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:285) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:251) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:220) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendServerLargeMessageChunk(ActiveMQSessionContext.java:460) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSendServer(ClientProducerImpl.java:388) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSend(ClientProducerImpl.java:330) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:266) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:128) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl$2.run(BridgeImpl.java:685) [artemis-server-1.1.0.SP18-redhat-1.jar:]
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103) [artemis-core-client-1.1.0.SP18-redhat-1.jar:]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]

Notes

This scenario only with one server (no cluster, just send/receive on one node) passes OK.
Message size is larger than server JVM memory. Artemis docs says following : "The only realistic limit to the size of a message that can be sent or consumed is the amount of disk space you have available".

Customer impact: If large message (~GB) will be sent to server and redistributed/load-balanced to another server in cluster then source server will crash on OutOfMemoryError leaving any service unavailable.

There is no known workaround.

Update with 7.1.0.DR12

The very same scenario now produces following error during message redistribution between cluster nodes. This is thrown on sending node.

17:52:31,171 WARN  [org.apache.activemq.artemis.utils.OrderedExecutorFactory] (Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@58d509f4-431458715)) failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488): io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 721420295, max: 734527488)
	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:624) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:578) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:686) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:675) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.allocate(PoolArena.java:221) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:372) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:120) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:284) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:265) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1036) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1044) [netty-all-4.1.5.Final-redhat-1.jar:4.1.5.Final-redhat-1]
	at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.writeBytes(ChannelBufferWrapper.java:570) [artemis-commons-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionContinuationMessage.encodeRest(SessionContinuationMessage.java:67) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.wireformat.SessionSendContinuationMessage.encodeRest(SessionSendContinuationMessage.java:96) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:291) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.PacketImpl.encode(PacketImpl.java:277) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:273) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:225) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.sendServerLargeMessageChunk(ActiveMQSessionContext.java:468) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSendServer(ClientProducerImpl.java:379) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.largeMessageSend(ClientProducerImpl.java:323) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:261) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:126) [artemis-core-client-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl$2.run(BridgeImpl.java:668) [artemis-server-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.3.002-redhat-1.jar:1.5.3.002-redhat-1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_65]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_65]
	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_65]

is blocked by

JBEAP-6398 OutOfDirectMemoryError raised from Netty

Closed

is cloned by

ENTMQBR-605 Streaming huge messages between cluster nodes causes java.lang.OutOfMemoryError: Direct buffer memory

Closed

cloned to: ARTEMIS-1098 Loading...

Details

Description

Update with 7.1.0.DR12

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates