Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10414

Deadlock in cluster test case with network failures

    XMLWordPrintable

Details

    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout f1860bef3e523ad4ef25499554cfb7508f464c53
groovy -DEAP_VERSION=7.1.0.DR17 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=ClusterTestCase#clusterTestWithNetworkFailures -DfailIfNoTests=false -Deap=7x | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout f1860bef3e523ad4ef25499554cfb7508f464c53 groovy -DEAP_VERSION=7.1.0.DR17 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ClusterTestCase#clusterTestWithNetworkFailures -DfailIfNoTests= false -Deap=7x | tee log
    • AMQ Sprint 1

    Description

      Scenario:

      • There are two servers in cluster.
      • Producer sends 10 000 messages to server 1.
      • Receiver receives messages from server 2.
      • During receiving of messages server 1 is several times suspended to cause expiration of timeouts.

      Expectation: Receiver receives all messages sent by producer.

      Reality: Receiver doesn't receive all messages. Cluster bridge is not reconnected because of deadlock.

      Customer impact: There is a risk of rising deadlock in cluster scenario. The test reveals that the deadlock blocks reconnection of cluster bridge. This issue may have serious impact on entire messaging service.

      Determination if it is regression is very difficult. The deadlock was hit in scenario with JDBC persistence store. However stack traces of threads causing the dead lock don't refer to JDBC. I think it is just easier to hit it with JDBC store because it is much more slower than file-based store.

      In [1] you can see stack traces of threads causing the deadlock. For entire thread dump see attachment.

      [1]

      "Thread-21 (ActiveMQ-client-global-threads-1513770209)":
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.getConsumerCount(QueueImpl.java:834)
	- waiting to lock <0x00000000d494d950> (a org.apache.activemq.artemis.core.server.impl.QueueImpl)
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:1314)
	- locked <0x00000000d4a071e0> (a org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl)
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.handleNotificationMessage(ClusterConnectionImpl.java:1029)
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.onMessage(ClusterConnectionImpl.java:1004)
	- locked <0x00000000d4a071e0> (a org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49)
	at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124)
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"Thread-8 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$3@6d0df3ab-2070751283)":
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl.disconnectBindings(ClusterConnectionImpl.java:1152)
	- waiting to lock <0x00000000d4a071e0> (a org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl)
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionImpl.disconnectRecord(ClusterConnectionImpl.java:1485)
	at org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge.fail(ClusterConnectionBridge.java:357)
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:645)
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.connectionFailed(BridgeImpl.java:601)
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:717)
	at org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:582)
	- locked <0x00000000d494e920> (a org.apache.activemq.artemis.core.server.cluster.impl.ClusterConnectionBridge)
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.handle(QueueImpl.java:2598)
	- eliminated <0x00000000d494d950> (a org.apache.activemq.artemis.core.server.impl.QueueImpl)
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1995)
	- locked <0x00000000d494d950> (a org.apache.activemq.artemis.core.server.impl.QueueImpl)
	at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1700(QueueImpl.java:101)
	at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2784)
	- locked <0x00000000d4a07790> (a org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner)
	at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:101)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

      Attachments

        Issue Links

          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-11000 Upgrade Artemis 1.5.5.jbossorg-001

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          is related to

          ARTEMIS-1140 Loading...

          Activity

            People

              rh-ee-ataylor Andy Taylor
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: