Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10475

IndexOutOfBoundsException during reading message if compacting is enabled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Blocker
    • None
    • 7.1.0.DR16
    • ActiveMQ
    • None
    • Regression
    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout 69c191fa8ff419d9c1fb1ac4ba1ceba6b35551ce
groovy -DEAP_VERSION=7.1.0.DR16 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=RemoteJcaRebalancingTestCase#testLoadBalancingOfInboundConnectionsToClusterOneServerRestart -DfailIfNoTests=false -Deap=7x | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 69c191fa8ff419d9c1fb1ac4ba1ceba6b35551ce groovy -DEAP_VERSION=7.1.0.DR16 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=RemoteJcaRebalancingTestCase#testLoadBalancingOfInboundConnectionsToClusterOneServerRestart -DfailIfNoTests= false -Deap=7x | tee log

    Description

      Scenario:

      • There are two servers called "JMS servers"
      • There are two servers called "MDB servers"
      • JMS servers are configured to be in cluster. They have destinations and hold all messages.
      • MDB servers have MDB applications which connect remotely via resource adapter to JMS servers.
      • All types of messages are sent to JMS server 1
      • MDB applications are deployed to MDB servers and they start to resending messages from InQueue to OutQueue
      • During the test all servers are restarted.
      • After restarts receiver is connected to JMS server 1 and receives messages from OutQueue

      Expectation: All messages sent to InQueue are received from OutQueue.
      Reality: Some messages are not received. During resending of messages from InQueue to OutQueue I see a lot of exceptions [1].

      Customer impact: In described scenario MDBs are not able to process messages, because messages were likely corrupted by compacting. However this is just hypothesis. Real root cause of this issue is unknown.

      I see the issue only if compacting is enabled. Probably it also depends on block size of the disc because I am not able hit it locally but I can hit it in the lab on NFS. Block size of NFS is parametrized. Similar symptoms was already reported in JBEAP-10062.

      This is regression against 7.1.0.DR11. It wasn't hit during DR12-DR15 testing because the test turned off compacting from some reason.

      During the test I can see also a lot of warnings [2] and [3].

      [1]

      09:03:01,066 ERROR [org.apache.activemq.artemis.ra] (Thread-19 (ActiveMQ-client-global-threads-1210222286)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException: readerIndex(17) + length(1) 
exceeds writerIndex(17): UnpooledDuplicatedByteBuf(ridx: 17, widx: 17, cap: 17, unwrapped: UnpooledUnsafeHeapByteBuf(ridx: 17, widx: 17, cap: 17))
        at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1395) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
        at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:687) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
        at io.netty.buffer.WrappedByteBuf.readByte(WrappedByteBuf.java:521) [netty-all-4.1.5.Final-redhat-2.jar:4.1.5.Final-redhat-2]
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112) [artemis-jms-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295)
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124) [artemis-core-client-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101) [artemis-commons-1.5.4.002-redhat-1.jar:1.5.4.002-redhat-1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_121]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_121]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_121]

      [2]

      03:37:30,868 WARN  [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142007: Can not find record 54 333 during compact replay

      [3]

      03:37:30,869 WARN  [org.apache.activemq.artemis.journal] (Thread-0 (ActiveMQ-IO-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$4@24d01472-1949610425)) AMQ142012: Couldnt find tx={0} to merge after compacting

      Attachments

        Issue Links

          duplicates

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-9937 Lost messages in cluster with server crashes

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-10454 Upgrade Artemis 1.5.4.jbossorg-003

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          is related to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-10062 Lost paged messages if journal is located on NFS

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-9937 Lost messages in cluster with server crashes

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Verified

          ARTEMIS-1114 Loading...

          Activity

            People

              rh-ee-ataylor Andy Taylor
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: