Uploaded image for project: 'WildFly WIP'
  1. WildFly WIP
  2. WFWIP-56

IndexOutOfBoundsException in JDBC HA scenario

XMLWordPrintable

    • Hide

      This issue is intermittent, it's necessary to run reproducer many times:

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/97/artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1531731315-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
      Show
      This issue is intermittent , it's necessary to run reproducer many times: git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/97/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1531731315-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log

      PR: https://github.com/wildfly/wildfly/pull/11355

      Scenario:

      • There are two Wildfly servers each containing two Artemis brokers
      • Artemis brokers creates two live-backup pairs. Each Wildfly instance contains one live and one backup
      • Second server has MDB which resends messages from InQueue to OutQueue
      • The second server is stopped and restarted

      After the second server is restarted, everything works correctly for a while. Failback performs successfully, MDB resends messages. However after some time the server starts to print many errors [1]. It looks like some messages where corrupted during failover/failback and cannot be delivered.

      The reproducer always hits the issue. The issue is not relevant to particular database, I hit it with Oracle12c and DB2 11.1. I also tried to reduce amount of messages and their size, but I still hit the issue. So it is not related to paging or large messages.

      [1]

      15:20:50,010 ERROR [org.apache.activemq.artemis.ra] (Thread-7 (ActiveMQ-client-global-threads)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:93) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:73) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112) [artemis-jms-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: