Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-29281

Corrupted large message after a server restart in the collocated HA cluster

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • 8.1.0.GA-CR4, 8.1.0.GA
    • 8.1.0.Beta
    • ActiveMQ
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Known Issue
    • Hide

      Steps to reproduce (issue is intermittent):

      git clone git@gitlab.cee.redhat.com:jbossqe-eap/messaging-testsuite.git messaging-testsuite
cd messaging-testsuite/scripts/


groovy -DEAP_ZIP_URL=<path_to_server_zip_file> PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/
mvn --batch-mode clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdown -Dsurefire.failIfNoSpecifiedTests=false -Deap7.clients.version=8.1739126675-SNAPSHOT -Deap7.org.jboss.qa.hornetq.apps.clients.version=8.1739126675-SNAPSHOT | tee log
      Show
      Steps to reproduce (issue is intermittent): git clone git@gitlab.cee.redhat.com:jbossqe-eap/messaging-testsuite.git messaging-testsuite cd messaging-testsuite/scripts/ groovy -DEAP_ZIP_URL=<path_to_server_zip_file> PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn --batch-mode clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdown -Dsurefire.failIfNoSpecifiedTests= false -Deap7.clients.version=8.1739126675-SNAPSHOT -Deap7.org.jboss.qa.hornetq.apps.clients.version=8.1739126675-SNAPSHOT | tee log

      There is corrupted large message after clean shutdown and restart of EAP 8.1.0 Beta CR4 server (ActiveMQ Artemis 2.38.0)  which is part of collocated HA cluster.

      Customer impact: In a collocated HA topology, if one of the EAP servers is shut down and restarted (a standard administrative operation) there is a possibility that one or more large messages (over 100KB by default) may become corrupted.

      Test Scenario:

      • start two EAP servers in colocated HA topology in cluster with InQueue and OutQueue
      • start producer to each of the servers and send 6000 messages to InQueue
      • deploy MDBs to 2nd node which consumes messages from InQueue and resends new message to OutQueue
      • wait until some messages are processed
      • shut down node-2
      • check if backup server on node-1 comes alive
      • start node-2 again
      • check failback of live on node-2
      • when all messages are processed, consume all messages from OutQueue

      Expected result: Receiver gets all messages which were sent.
      Actual result: One or more messages are not delivered into OutQueue. Node-2 throws:

      11:47:49,586 ERROR [org.apache.activemq.artemis.ra.ActiveMQRALogger] (Thread-14 (ActiveMQ-client-global-threads)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException: Error reading in simpleString, length=16777216 is greater than readableBytes=413159
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:255)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:243)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:248)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.collections.TypedProperties.decode(TypedProperties.java:510)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.collections.TypedProperties.decode(TypedProperties.java:581)
        at org.apache.activemq.artemis.client@2.38.0//org.apache.activemq.artemis.reader.MapMessageUtil.readBodyMap(MapMessageUtil.java:46)
        at org.apache.activemq.artemis.client@2.38.0//org.apache.activemq.artemis.jms.client.ActiveMQMapMessage.doBeforeReceive(ActiveMQMapMessage.java:327)
        at org.apache.activemq.artemis.ra@2.38.0//org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:313)
        at org.apache.activemq.artemis.client@2.38.0//org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:971)
        at org.apache.activemq.artemis.client@2.38.0//org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1128)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at org.apache.activemq.artemis.commons@2.38.0//org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

      anytime it tries to receive corrupted message from InQueue.

              ehugonne1@redhat.com Emmanuel Hugonnet
              mnovak1@redhat.com Miroslav Novak
              Miroslav Novak Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: