Loading...

Details

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: 14.0.0.Final
Affects Version/s: 14.0.0.Beta2
Component/s: Clustering
Labels:
None

Steps to Reproduce:
Hide

Note: The reproduce is not 100%. You have to run it several times to hit the issue.

git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/185/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ClusteredSingletonMDBTestCase#clusterMDBSigletonTestDescriptorsShutdown -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1533805308-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
Show
Note: The reproduce is not 100%. You have to run it several times to hit the issue. git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/185/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ClusteredSingletonMDBTestCase#clusterMDBSigletonTestDescriptorsShutdown -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1533805308-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log
Affects Testing:

Regression
Git Pull Request:
https://github.com/wildfly/wildfly/pull/11545

Description

Scenario

There are two Wildfly servers configured in cluster
Both servers have deployment with clustered singleton MDB
Start both servers
Check that MDB is active only on one server
Shutdown server with active MDB
Check that MDB get activated on the second server.

Sometimes happens that MDB doesn't get activated on the second server. The test waits 60 seconds for it. Always when this issue happens, I can see following exception in the log of the second server which is logged after the first server is shutdown.

The issue was hit with WildFly master built on 9th August (6675410).
Blocker priority was set because it is regression against WF 13.

09:12:45,274 WARN  [org.wildfly.clustering.server] (DistributedSingletonService - 1) WFLYCLSV0012: Failed to notify server/default service provider registration listener of new providers: [node-2]: java.util.con
current.CompletionException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@151e803d rejected from java.util.concurrent.ThreadPoolExecutor@46fff41f[Terminated, pool size = 
0, active threads = 0, queued tasks = 0, completed tasks = 2]
        at java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375) [rt.jar:1.8.0_171]
        at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934) [rt.jar:1.8.0_171]
        at org.jgroups.blocks.UnicastRequest.lambda$join$0(UnicastRequest.java:110) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.blocks.UnicastRequest.around(UnicastRequest.java:137) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.blocks.UnicastRequest.join(UnicastRequest.java:110) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.wildfly.clustering.server.singleton.AbstractDistributedSingletonService.providersChanged(AbstractDistributedSingletonService.java:121) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.B
eta2-SNAPSHOT]
        at org.wildfly.clustering.server.provider.CacheServiceProviderRegistry.lambda$modified$5(CacheServiceProviderRegistry.java:285) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_171]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
        at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47) [wildfly-clustering-service-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
        at org.jboss.threads.JBossThread.run(JBossThread.java:485) [jboss-threads-2.3.2.Final.jar:2.3.2.Final]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@151e803d rejected from java.util.concurrent.ThreadPoolExecutor@46fff41f[Terminated, pool size = 0, active threads 
= 0, queued tasks = 0, completed tasks = 2]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) [rt.jar:1.8.0_171]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [rt.jar:1.8.0_171]
        at org.wildfly.clustering.server.dispatcher.ChannelCommandDispatcherFactory.handle(ChannelCommandDispatcherFactory.java:143) [wildfly-clustering-server-14.0.0.Beta2-SNAPSHOT.jar:14.0.0.Beta2-SNAPSHOT]
        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:383) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:356) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:307) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:582) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.JChannel.up(JChannel.java:816) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:893) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FORK.up(FORK.java:130) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FRAG3.up(FRAG3.java:171) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:351) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FlowControl.up(FlowControl.java:359) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:873) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:375) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:130) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:203) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:253) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.MERGE3.up(MERGE3.java:280) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.Discovery.up(Discovery.java:269) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.protocols.TP.passMessageUp(TP.java:1248) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87) [jgroups-4.0.13.Final.jar:4.0.13.Final]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
        at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]

Attachments

Issue Links

causes

WFLY-10583 SingletonPolicyServiceTestCase fails intermittently

Closed

Clustered singleton MDB doesn't get activated after master node is shutdown

Details

Description

Attachments

Issue Links

Activity

People

Dates