Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-1617

[Artemis 2.x upgrade] Unexptected crash of server in SOAK test

XMLWordPrintable

    • Hide
      export DISABLE_TRACE_LOGS=false
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      git checkout 1caebfb776230ee2a3a1857535a5ac86e6e31304
      groovy -DdisableTraceLogs  -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      
      cd ../jboss-hornetq-testsuite/
      
      mvn -Deap7.clients.version=7.1521816809-SNAPSHOT -Dgroups=category.Soak -Dsurefire.timeout=193600 -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521816809-SNAPSHOT clean install
      
      Show
      export DISABLE_TRACE_LOGS= false git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 1caebfb776230ee2a3a1857535a5ac86e6e31304 groovy -DdisableTraceLogs -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn -Deap7.clients.version=7.1521816809-SNAPSHOT -Dgroups=category.Soak -Dsurefire.timeout=193600 -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521816809-SNAPSHOT clean install

      After ~13 hours there is unexpected crash of one server in SOAK test. There is no error/warning in the logs.

      Test Scenario:

      • Start 2 servers
      • Client sends messages to input queue. Messages then go through:
      • One server to another through MDB reading and sending them from remote container through resource adapter
      • Messages are forwarded from one server to another over JMS bridge and back over Core bridge
      • Messages have JMSReplyTo defined with a temporary queue, that is filled with responses for the client
      • Messages are read from the destination with stateless EJB and sent back to clients
      • Client reads the messages after the pass through all the soak modules.

      Pass Criteria: In the last step receiver consumes all messages sent by producer.

      Actual Result:
      After ~13 hours 1st server suddenly crashes. There is no error/warning in server logs.

      Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_with_prefix (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)

        1. bridge_issue_node2.log
          112 kB
        2. bridge_issue.log
          495 kB
        3. serverlogs.zip
          1.77 MB

            mtaylor1@redhat.com Martyn Taylor (Inactive)
            mtoth@redhat.com Michal Toth
            Michal Toth Michal Toth
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 2 days
                2d
                Remaining:
                Remaining Estimate - 2 days
                2d
                Logged:
                Time Spent - Not Specified
                Not Specified