Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-1617

[Artemis 2.x upgrade] Unexptected crash of server in SOAK test

    XMLWordPrintable

Details

    • Hide 
      export DISABLE_TRACE_LOGS=false
git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout 1caebfb776230ee2a3a1857535a5ac86e6e31304
groovy -DdisableTraceLogs  -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn -Deap7.clients.version=7.1521816809-SNAPSHOT -Dgroups=category.Soak -Dsurefire.timeout=193600 -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521816809-SNAPSHOT clean install
      Show
      export DISABLE_TRACE_LOGS= false git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 1caebfb776230ee2a3a1857535a5ac86e6e31304 groovy -DdisableTraceLogs -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/273//artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn -Deap7.clients.version=7.1521816809-SNAPSHOT -Dgroups=category.Soak -Dsurefire.timeout=193600 -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1521816809-SNAPSHOT clean install

    Description

      After ~13 hours there is unexpected crash of one server in SOAK test. There is no error/warning in the logs.

      Test Scenario:

      • Start 2 servers
      • Client sends messages to input queue. Messages then go through:
      • One server to another through MDB reading and sending them from remote container through resource adapter
      • Messages are forwarded from one server to another over JMS bridge and back over Core bridge
      • Messages have JMSReplyTo defined with a temporary queue, that is filled with responses for the client
      • Messages are read from the destination with stateless EJB and sent back to clients
      • Client reads the messages after the pass through all the soak modules.

      Pass Criteria: In the last step receiver consumes all messages sent by producer.

      Actual Result:
      After ~13 hours 1st server suddenly crashes. There is no error/warning in server logs.

      Issue was hit with Artemis 2.5.0 with https://github.com/jmesnil/wildfly/tree/WFLY-9407_upgrade_artemis_2.4.0_with_prefix (commit 51dd8102f103ccb0470a3cfc8713d3f9bdb1b65d)

      Attachments

        1. bridge_issue_node2.log
          112 kB
        2. bridge_issue.log
          495 kB
        3. serverlogs.zip
          1.77 MB

        Issue Links

          blocks

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. WFLY-10320 Upgrade artemis from 1.5.x to 2.x.x

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          clones

          Bug - A problem that impairs or prevents the functions of the product. WFWIP-28 [Artemis 2.x upgrade] Unexptected crash of server in SOAK test

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              mtoth@redhat.com Michal Toth
              Michal Toth Michal Toth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 2 days
                  2d
                  Remaining:
                  Remaining Estimate - 2 days
                  2d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified