Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-5806

IllegalStateException during failover of HA Singleton deployment

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor Minor
    • 10.0.0.CR5
    • 10.0.0.CR4
    • Clustering
    • None

      Seen in failover tests - HA Singleton deployment scenarios - no matter what failover type was used (graceful shutdown, jvmkill, undeploy).

      When the node is elected to operate as the singleton provider, sometimes this INFO message is logged: 

      INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead.

      Then the redeploy will fail:

      [JBossINF] [0m[33m05:41:07,118 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)

      Then these events follows:

      • This node will no longer operate as the singleton provider
      • immediately after that the node is re-elected as the singleton provider and the deployment starts successfully

      See the full stacktrace:

      [JBossINF] [0m[0m05:41:07,109 INFO  [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:07,112 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-singleton-jbossall.ear, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-passivating.war, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-ejb.jar, performing full redeploy instead.
[JBossINF] [0m[0m05:41:07,114 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0070: Deployment restart detected for deployment clusterbench-ee7-web-default.war, performing full redeploy instead.
[JBossINF] [0m[33m05:41:07,118 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-passivating.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[33m05:41:07,117 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-7) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-web-default.war".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[33m05:41:07,116 WARN  [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000004: Failure during stop of service jboss.deployment.subunit."clusterbench-ee7-singleton-jbossall.ear"."clusterbench-ee7-ejb.jar".FIRST_MODULE_USE.service: java.lang.IllegalStateException
[JBossINF] 	at org.jboss.msc.value.InjectedValue.getValue(InjectedValue.java:47)
[JBossINF] 	at org.jboss.as.server.deployment.DeploymentUnitPhaseService.stop(DeploymentUnitPhaseService.java:225)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] 
[JBossINF] [0m[0m05:41:07,150 INFO  [org.wildfly.clustering.server] (OOB-19,ee,perf19) WFLYCLSV0002: This node will no longer operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:07,170 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000080: Disconnecting JGroups channel hibernate
[JBossINF] [0m[0m05:41:07,171 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000082: Stopping the RpcDispatcher for channel hibernate
[JBossINF] [0m[0m05:41:07,171 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000080: Disconnecting JGroups channel web
[JBossINF] [0m[0m05:41:07,172 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-7) ISPN000082: Stopping the RpcDispatcher for channel web
[JBossINF] [0m[0m05:41:07,172 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-ejb.jar) in 58ms
[JBossINF] [0m[0m05:41:07,173 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb
[JBossINF] [0m[0m05:41:07,173 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb
[JBossINF] [0m[0m05:41:07,176 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-default.war) in 62ms
[JBossINF] [0m[0m05:41:07,175 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0208: Stopped subdeployment (runtime-name: clusterbench-ee7-web-passivating.war) in 61ms
[JBossINF] [0m[0m05:41:07,184 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0028: Stopped deployment clusterbench-ee7-singleton-jbossall.ear (runtime-name: clusterbench-ee7-singleton-jbossall.ear) in 70ms
[JBossINF] [0m[0m05:41:07,186 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) WFLYSRV0027: Starting deployment of "clusterbench-ee7-singleton-jbossall.ear" (runtime-name: "clusterbench-ee7-singleton-jbossall.ear")
[JBossINF] [0m[0m05:41:07,187 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0003: Stopped default cache from server container
[JBossINF] [0m[0m05:41:07,190 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000080: Disconnecting JGroups channel server
[JBossINF] [0m[0m05:41:07,190 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-2) ISPN000082: Stopping the RpcDispatcher for channel server
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-passivating.war")
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-web-default.war")
[JBossINF] [0m[0m05:41:07,232 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee7-ejb.jar")
[JBossINF] [0m[33m05:41:07,294 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-8) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-8) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-default.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.private] (MSC service thread 1-1) WFLYSRV0018: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using a private module ("org.infinispan:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[33m05:41:07,295 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-1) WFLYSRV0019: Deployment "deployment.clusterbench-ee7-singleton-jbossall.ear.clusterbench-ee7-web-passivating.war" is using an unsupported module ("org.jgroups:main") which may be changed or removed in future versions without notice.
[JBossINF] [0m[0m05:41:07,423 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000078: Starting JGroups channel server
[JBossINF] [0m[0m05:41:07,424 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000094: Received new cluster view for channel server: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,424 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000079: Channel server local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,426 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000078: Starting JGroups channel web
[JBossINF] [0m[0m05:41:07,427 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000094: Received new cluster view for channel web: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,427 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000079: Channel web local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,442 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000078: Starting JGroups channel ejb
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000094: Received new cluster view for channel ejb: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000079: Channel ejb local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,444 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000078: Starting JGroups channel hibernate
[JBossINF] [0m[0m05:41:07,445 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000094: Received new cluster view for channel hibernate: [perf21|13] (4) [perf21, perf18, perf20, perf19]
[JBossINF] [0m[0m05:41:07,446 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-6) ISPN000079: Channel hibernate local address is perf19, physical addresses are [172.19.1.3:55200]
[JBossINF] [0m[0m05:41:07,510 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 82) WFLYCLINF0002: Started default cache from server container
[JBossINF] [0m[0m05:41:10,357 INFO  [org.wildfly.clustering.server] (OOB-20,ee,perf19) WFLYCLSV0001: This node will now operate as the singleton provider of the jboss.deployment.unit."clusterbench-ee7-singleton-jbossall.ear".FIRST_MODULE_USE service
[JBossINF] [0m[0m05:41:10,362 INFO  [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-singleton-jbossall.ear
[JBossINF] [0m[0m05:41:10,384 INFO  [org.jboss.weld.deployer] (MSC service thread 1-6) WFLYWELD0003: Processing weld deployment clusterbench-ee7-ejb.jar
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatelessSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:module/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatelessSBImpl
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatelessSBImpl
[JBossINF] 	java:module/RemoteStatelessSBImpl
[JBossINF] 
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatefulSBImpl' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:module/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:jboss/exported/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl!org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/RemoteStatefulSBImpl
[JBossINF] 	java:app/clusterbench-ee7-ejb/RemoteStatefulSBImpl
[JBossINF] 	java:module/RemoteStatefulSBImpl
[JBossINF] 
[JBossINF] [0m[0m05:41:10,387 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-6) WFLYEJB0473: JNDI bindings for session bean named 'LocalStatefulSB' in deployment unit 'subdeployment "clusterbench-ee7-ejb.jar" of deployment "clusterbench-ee7-singleton-jbossall.ear"' are as follows:
[JBossINF] 
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:module/LocalStatefulSB!org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB
[JBossINF] 	java:global/clusterbench-ee7-singleton-jbossall/clusterbench-ee7-ejb/LocalStatefulSB
[JBossINF] 	java:app/clusterbench-ee7-ejb/LocalStatefulSB
[JBossINF] 	java:module/LocalStatefulSB
[JBossINF]

      Link:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-singleton-deployment-jvmkill-random-election-policy/5/console-perf19/

              pferraro@redhat.com Paul Ferraro
              mvinkler1@redhat.com Michal Vinkler
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: