Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-10814

Clustered singleton MDB doesn't get activated after master node is shutdown

    XMLWordPrintable

Details

    • Hide

      Note: The reproduce is not 100%. You have to run it several times to hit the issue.

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/185/artifact/jboss-eap.zip PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      
      cd ../jboss-hornetq-testsuite/
      
      mvn clean test -Dtest=ClusteredSingletonMDBTestCase#clusterMDBSigletonTestDescriptorsShutdown -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1533805308-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
      
      Show
      Note: The reproduce is not 100%. You have to run it several times to hit the issue. git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/185/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ClusteredSingletonMDBTestCase#clusterMDBSigletonTestDescriptorsShutdown -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1533805308-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log
    • Regression

    Description

      Scenario

      • There are two Wildfly servers configured in cluster
      • Both servers have deployment with clustered singleton MDB
      • Start both servers
      • Check that MDB is active only on one server
      • Shutdown server with active MDB
      • Check that MDB get activated on the second server.

      Sometimes happens that MDB doesn't get activated on the second server. The test waits 60 seconds for it. Always when this issue happens, I can see following exception in the log of the second server which is logged after the first server is shutdown.

      The issue was hit with WildFly master built on 9th August (6675410).
      Blocker priority was set because it is regression against WF 13.

      09:12:45,274 WARN  [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0012: Failed to notify server/default service provider registration listener of new providers: [node-2]: java.util.con
      current.CompletionException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@151e803d rejected from java.util.concurrent.ThreadPoolExecutor@46fff41f[Terminated, pool size = 
      0, active threads = 0, queued tasks = 0, completed tasks = 2]
              at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375) [rt.jar:1.8.0_171]
              at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934) [rt.jar:1.8.0_171]
              at org.jgroups.blocks.UnicastRequest.lambda$join$0(UnicastRequest.java:110) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.blocks.UnicastRequest.around(UnicastRequest.java:137) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.blocks.UnicastRequest.join(UnicastRequest.java:110) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.wildfly.clustering.server.singleton.AbstractDistributedSingletonService.providersChanged(AbstractDistributedSingletonService.java:121) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.B
      eta2-SNAPSHOT]
              at org.wildfly.clustering.server.provider.CacheServiceProviderRegistry.lambda$modified$5(CacheServiceProviderRegistry.java:285) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_171]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
              at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) [wildfly-clustering-service-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
              at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
      Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@151e803d rejected from java.util.concurrent.ThreadPoolExecutor@46fff41f[Terminated, pool size = 0, active threads 
      = 0, queued tasks = 0, completed tasks = 2]
              at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) [rt.jar:1.8.0_171]
              at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [rt.jar:1.8.0_171]
              at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcherFactory.handle(ChannelCommandDispatcherFactory.java:143) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
              at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:383) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:356) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:307) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:582) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.JChannel.up(JChannel.java:816) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FORK.up(FORK.java:130) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FRAG3.up(FRAG3.java:171) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:351) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:359) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:873) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:375) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:130) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:203) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:253) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.MERGE3.up(MERGE3.java:280) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.Discovery.up(Discovery.java:269) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.protocols.TP.passMessageUp(TP.java:1248) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) [jgroups-4.0.13.Final.jar:4.0.13.Final]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
              at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
      

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: