Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-2473

[EAP] NettyColocatedClusterFailoverTestCase possible deadlock

    XMLWordPrintable

Details

    • Hide

      mvn -f eap-testsuite/jboss-hornetq-testsuite/pom.xml -DEAP_VERSION=7.1556098479-SNAPSHOT -DEAP_ZIP_URL=https://mw-messaging-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/332//artifact/jboss-eap.zip -DARTEMIS_REPO= -DTESTSUITE_REPO=git@bitbucket.org:msgqe/eap-testsuite-mirror.git -DTESTSUITE_BRANCH=rvais_changes -Dgroups=category.FailoverColocatedCluster -Dmaven.test.failure.ignore=true -DexcludedGroups=category.scope.Integration -DfailIfNoTests=false -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1556098479-SNAPSHOT -Dmaven.repo.local=/home/buildadm/workspace/early-testing-messaging-weekly-tests-ha-failover-colocated-cluster/.repository clean install

      Show
      mvn -f eap-testsuite/jboss-hornetq-testsuite/pom.xml -DEAP_VERSION=7.1556098479-SNAPSHOT -DEAP_ZIP_URL= https://mw-messaging-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/early-testing-messaging-prepare/332//artifact/jboss-eap.zip -DARTEMIS_REPO= -DTESTSUITE_REPO=git@bitbucket.org:msgqe/eap-testsuite-mirror.git -DTESTSUITE_BRANCH=rvais_changes -Dgroups=category.FailoverColocatedCluster -Dmaven.test.failure.ignore=true -DexcludedGroups=category.scope.Integration -DfailIfNoTests=false -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1556098479-SNAPSHOT -Dmaven.repo.local=/home/buildadm/workspace/early-testing-messaging-weekly-tests-ha-failover-colocated-cluster/.repository clean install

    Description

      Concurrency issue with the exceptions on the locks which makes me suspect deadlock or live lock issue. Snippet of the log below, partial and full logs attached as extra files. Direct link to Jenkins

      Unable to find source-code formatter for language: stacktrace. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
      Stack trace of thread: Thread[Thread-3 (ActiveMQ-client-global-scheduled-threads),5,FailOnTimeoutGroup]
      ---sun.misc.Unsafe.park(Native Method)
      ---java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      ---java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      ---java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      ---java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      ---java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
      ---java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
      ---java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      ---org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      ---------------------------------------------
      Stack trace of thread: Thread[Thread-100 (ActiveMQ-client-global-threads),5,FailOnTimeoutGroup]
      ---sun.misc.Unsafe.park(Native Method)
      ---java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      ---java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
      ---java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
      ---java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
      ---org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.waitOnLatch(ActiveMQClientProtocolManager.java:136)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:830)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:747)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:617)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:507)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.access$600(ClientSessionFactoryImpl.java:73)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1222)
      ---org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.callFailureListeners(AbstractRemotingConnection.java:75)
      ---org.apache.activemq.artemis.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:219)
      ---org.apache.activemq.artemis.spi.core.protocol.AbstractRemotingConnection.fail(AbstractRemotingConnection.java:218)
      ---org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1011)
      ---org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
      ---org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
      ---org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66)
      ---org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$186/1192073875.run(Unknown Source)
      ---java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      ---java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      ---org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
      ---------------------------------------------
      ...
      

      Attachments

        1. consoleText.log
          29.12 MB
        2. PartialLog.log
          45 kB

        Issue Links

          Activity

            People

              dbruscin Domenico Francesco Bruscino
              rvais Roman Vais
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: