Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10475

IndexOutOfBoundsException during reading message if compacting is enabled

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • None
    • 7.1.0.DR16
    • ActiveMQ
    • None
    • Regression
    • Hide
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      git checkout 69c191fa8ff419d9c1fb1ac4ba1ceba6b35551ce
      groovy -DEAP_VERSION=7.1.0.DR16 PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      
      cd ../jboss-hornetq-testsuite/
      
      mvn clean test -Dtest=RemoteJcaRebalancingTestCase#testLoadBalancingOfInboundConnectionsToClusterOneServerRestart -DfailIfNoTests=false -Deap=7x | tee log
      
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 69c191fa8ff419d9c1fb1ac4ba1ceba6b35551ce groovy -DEAP_VERSION=7.1.0.DR16 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=RemoteJcaRebalancingTestCase#testLoadBalancingOfInboundConnectionsToClusterOneServerRestart -DfailIfNoTests= false -Deap=7x | tee log

      Scenario:

      • There are two servers called "JMS servers"
      • There are two servers called "MDB servers"
      • JMS servers are configured to be in cluster. They have destinations and hold all messages.
      • MDB servers have MDB applications which connect remotely via resource adapter to JMS servers.
      • All types of messages are sent to JMS server 1
      • MDB applications are deployed to MDB servers and they start to resending messages from InQueue to OutQueue
      • During the test all servers are restarted.
      • After restarts receiver is connected to JMS server 1 and receives messages from OutQueue

      Expectation: All messages sent to InQueue are received from OutQueue.
      Reality: Some messages are not received. During resending of messages from InQueue to OutQueue I see a lot of exceptions [1].

      Customer impact: In described scenario MDBs are not able to process messages, because messages were likely corrupted by compacting. However this is just hypothesis. Real root cause of this issue is unknown.

      I see the issue only if compacting is enabled. Probably it also depends on block size of the disc because I am not able hit it locally but I can hit it in the lab on NFS. Block size of NFS is parametrized. Similar symptoms was already reported in JBEAP-10062.

      This is regression against 7.1.0.DR11. It wasn't hit during DR12-DR15 testing because the test turned off compacting from some reason.

      During the test I can see also a lot of warnings [2] and [3].

      [1]

      09:03:01,066 ERROR [org.apache.activemq.artemis.ra] (Thread-19 (ActiveMQ-client-global-threads-1210222286)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException: readerIndex(17) + length(1) 
      exceeds writerIndex(17): UnpooledDuplicatedByteBuf(ridx: 17, widx: 17, cap: 17, unwrapped: UnpooledUnsafeHeapByteBuf(ridx: 17, widx: 17, cap: 17))
              at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1395) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
              at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:687) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
              at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:521) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
              at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112) [artemis-jms-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295)
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]
      

      [2]

      03:37:30,868 WARN  [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142007: Can not find record 54 333 during compact replay
      

      [3]

      03:37:30,869 WARN  [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142012: Couldnt find tx={0} to merge after compacting
      

              rh-ee-ataylor Andy Taylor
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: