Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-11184

EAP doesn't stop if JMS bridge uses connection factory with ha=false and call-failover-timeout=-1

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Critical Critical
    • None
    • 7.1.0.DR19, 7.1.0.ER1, 7.1.0.CR1
    • ActiveMQ
    • None
    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout 5b21ed48fe285d92e107fd6a149bef6fe856b000
groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=JMSBridgeTestCase#testSourceServerShutdownWhenTargetIsOff -DfailIfNoTests=false -Deap=7x | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 5b21ed48fe285d92e107fd6a149bef6fe856b000 groovy -DEAP_VERSION=7.1.0.DR19 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=JMSBridgeTestCase#testSourceServerShutdownWhenTargetIsOff -DfailIfNoTests= false -Deap=7x | tee log

      Scenario

      • There are two EAP servers with JMS bridge configured to resend messages from server 1 to server 2
      • Servers are stopped in order server 2, server 1

      Expectation: Both servers are properly stopped.

      Reality: Server 1 doesn't stop in 2 minutes timeout.

      Customer impact: The issue may affect user experience or break customer's automation.

      Based on the thread dump [1] stopping of EAP hangs in ChannelImpl.waitForFailOver method. JMSBridge tries to do failover. Since call-failover-timeout=-1 it waits infinitely. It should detect that the server is stopping and give up trying to do failover.

      I also found out that the issue occurs only if target's connection factory has ha=false. If ha=true, both servers are properly stopped.

      [1]

      "ServerService Thread Pool -- 79" #158 prio=5 os_prio=0 tid=0x00007f7a5c027800 nid=0x5209 waiting on condition [0x00007f79cc7aa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000f7d9e520> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.waitForFailOver(ChannelImpl.java:250)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:353)
        - locked <0x00000000f7d9e548> (a java.lang.Object)
        at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:315)
        at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQSessionContext.xaEnd(ActiveMQSessionContext.java:384)
        at org.apache.activemq.artemis.core.client.impl.ClientSessionImpl.end(ClientSessionImpl.java:1173)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.doEnd(TransactionImple.java:1089)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.endAssociation(TransactionImple.java:1060)
        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.endAssociation(XAResourceRecord.java:1287)
        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:313)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3023)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3002)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1674)
        - locked <0x00000000f7d9d5c8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:124)
        at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:215)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.rollback(TransactionImple.java:284)
        at org.wildfly.transaction.client.provider.jboss.JBossLocalTransactionProvider$Entry.rollbackLocal(JBossLocalTransactionProvider.java:317)
        at org.wildfly.transaction.client.provider.jboss.JBossJTALocalTransactionProvider.rollbackLocal(JBossJTALocalTransactionProvider.java:116)
        at org.wildfly.transaction.client.LocalTransaction.rollback(LocalTransaction.java:88)
        at org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl.stop(JMSBridgeImpl.java:501)
        - locked <0x00000000d07b6860> (a org.apache.activemq.artemis.jms.bridge.impl.JMSBridgeImpl)
        at org.wildfly.extension.messaging.activemq.jms.bridge.JMSBridgeService$2.run(JMSBridgeService.java:127)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
        at org.jboss.threads.JBossThread.run(JBossThread.java:320)

   Locked ownable synchronizers:
        - <0x00000000f7c73c20> (a java.util.concurrent.ThreadPoolExecutor$Worker)

        1. JBEAP-11184.patch
          5 kB
          Jeff Mesnil
        2. node-1-thread-dump-when-killed-shutdown-sequence.txt
          112 kB
          Erich Duda

              mstefank Martin Stefanko
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: