Scenario:
- There are two servers called "JMS servers"
- There are two servers called "MDB servers"
- JMS servers are configured to be in cluster. They have destinations and hold all messages.
- MDB servers have MDB applications which connect remotely via resource adapter to JMS servers.
- All types of messages are sent to JMS server 1
- MDB applications are deployed to MDB servers and they start to resending messages from InQueue to OutQueue
- During the test all servers are restarted.
- After restarts receiver is connected to JMS server 1 and receives messages from OutQueue
Expectation: All messages sent to InQueue are received from OutQueue.
Reality: Some messages are not received. During resending of messages from InQueue to OutQueue I see a lot of exceptions [1].
Customer impact: In described scenario MDBs are not able to process messages, because messages were likely corrupted by compacting. However this is just hypothesis. Real root cause of this issue is unknown.
I see the issue only if compacting is enabled. Probably it also depends on block size of the disc because I am not able hit it locally but I can hit it in the lab on NFS. Block size of NFS is parametrized. Similar symptoms was already reported in JBEAP-10062.
This is regression against 7.1.0.DR11. It wasn't hit during DR12-DR15 testing because the test turned off compacting from some reason.
During the test I can see also a lot of warnings [2] and [3].
[1]
09:03:01,066 ERROR [org.apache.activemq.artemis.ra] (Thread-19 (ActiveMQ-client-global-threads-1210222286)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException: readerIndex(17) + length(1) exceeds writerIndex(17): UnpooledDuplicatedByteBuf(ridx: 17, widx: 17, cap: 17, unwrapped: UnpooledUnsafeHeapByteBuf(ridx: 17, widx: 17, cap: 17)) at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1395) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2] at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:687) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2] at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:521) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2] at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112) [artemis-jms-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295) at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
[2]
03:37:30,868 WARN [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142007: Can not find record 54 333 during compact replay
[3]
03:37:30,869 WARN [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142012: Couldnt find tx={0} to merge after compacting
- duplicates
-
JBEAP-9937 Lost messages in cluster with server crashes
- Closed
- is incorporated by
-
JBEAP-10454 Upgrade Artemis 1.5.4.jbossorg-003
- Closed
- is related to
-
JBEAP-10062 Lost paged messages if journal is located on NFS
- Closed
- relates to
-
JBEAP-9937 Lost messages in cluster with server crashes
- Closed
-
ARTEMIS-1114 Loading...