Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: 7.1.0.DR16
Affects Version/s: 7.0.0.ER2 (Beta), 7.1.0.DR13
Component/s: Clustering
Labels:
None

CDW devel_ack:
CDW docs_ack:
CDW pm_ack:
CDW qa_ack:
CDW release:
Target Release:

7.1.0.GA
Git Pull Request:
https://github.com/jbossas/jboss-eap7/pull/140, https://github.com/jbossas/jboss-eap7/pull/1535

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Seen in failover tests - HA Singleton deployment scenarios - no matter what failover type was used (graceful shutdown, jvmkill, undeploy).

When the node is elected to operate as the singleton provider, sometimes this INFO message is logged:

INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead.

Then the redeploy will fail:

[JBossINF] [0m[33m05:41:07,118 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)

Then these events follows:

This node will no longer operate as the singleton provider
immediately after that the node is re-elected as the singleton provider and the deployment starts successfully

See the full stacktrace:

[JBossINF] [0m[0m05:41:07,109 INFO  [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:07,112 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-passivating.war, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-ejb.jar, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-default.war, performing full redeploy instead.
[JBossINF] [0m[33m05:41:07,118 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[33m05:41:07,117 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-default.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[33m05:41:07,116 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-ejb.jar".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[0m05:41:07,150 INFO  [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0002: This node will no longer operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:07,170 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel hibernate
[JBossINF] [0m[0m05:41:07,171 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher for channel hibernate
[JBossINF] [0m[0m05:41:07,171 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel web
[JBossINF] [0m[0m05:41:07,172 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web
[JBossINF] [0m[0m05:41:07,172 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-ejb.jar) in 58ms
[JBossINF] [0m[0m05:41:07,173 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m05:41:07,173 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb
[JBossINF] [0m[0m05:41:07,176 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-default.war) in 62ms
[JBossINF] [0m[0m05:41:07,175 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-passivating.war) in 61ms
[JBossINF] [0m[0m05:41:07,184 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment clusterbench-ee7-singleton-jbossall.ear (runtime-name: clusterbench-ee7-singleton-jbossall.ear) in 70ms
[JBossINF] [0m[0m05:41:07,186 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "clusterbench-ee7-singleton-jbossall.ear" (runtime-name: "clusterbench-ee7-singleton-jbossall.ear")
[JBossINF] [0m[0m05:41:07,187 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0003: Stopped default cache from server container
[JBossINF] [0m[0m05:41:07,190 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel server
[JBossINF] [0m[0m05:41:07,190 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel server
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war")
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war")
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar")
[JBossINF] [0m[33m05:41:07,294 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-8) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-8) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-1) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[0m05:41:07,423 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000078: Starting JGroups channel server
[JBossINF] [0m[0m05:41:07,424 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view for channel server: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,424 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000079: Channel server local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,426 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel web
[JBossINF] [0m[0m05:41:07,427 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel web: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,427 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel web local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,442 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel hibernate
[JBossINF] [0m[0m05:41:07,445 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel hibernate: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,446 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel hibernate local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,510 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0002: Started default cache from server container
[JBossINF] [0m[0m05:41:10,357 INFO  [org.wildfly.clustering.server] (OOB-20,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:10,362 INFO  [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-singleton-jbossall.ear
[JBossINF] [0m[0m05:41:10,384 INFO  [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-ejb.jar
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatelessSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:module/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl
[JBossINF] 	java:module/RemoteStatelessSBImpl
[JBossINF] 
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatefulSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:module/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl
[JBossINF] 	java:module/RemoteStatefulSBImpl
[JBossINF] 
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'LocalStatefulSB' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:module/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/LocalStatefulSB
[JBossINF] 	java:module/LocalStatefulSB
[JBossINF]

Link:
http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singleton-deployment-jvmkill-random-election-policy/5/console-perf19/

causes

JBEAP-9360 Singleton deployment re-election is not working properly

Closed

is cloned by

WFLY-5806 IllegalStateException during failover of HA Singleton deployment

Closed

relates to

JBEAP-2260 (7.2.0) IndexOutOfBoundsException after stopping deployment of HA Singleton deployment

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates