Uploaded image for project: 'Application Server 7'
  1. Application Server 7
  2. AS7-4144

Failures in cluster cause significat disruption (IllegalStateException, "Rolling back to cache view Y, but last committed view is Z", TOs)

    XMLWordPrintable

Details

    Description

      Negative failover scenarios (server crashing) cause significant distruptions in the cluster.

      [JBossINF] 17:19:13,423 INFO  [org.jboss.weld.deployer] (MSC service thread 1-6) JBAS016002: Processing weld deployment clusterbench-ee6-web.war
[JBossINF] 17:19:13,433 INFO  [org.jboss.weld.deployer] (MSC service thread 1-15) JBAS016005: Starting Services for CDI deployment: clusterbench-ee6.ear
[JBossINF] 17:19:13,527 INFO  [org.jboss.weld.Version] (MSC service thread 1-15) WELD-000900 2012-02-22 16:46
[JBossINF] 17:19:13,552 INFO  [org.jboss.weld.deployer] (MSC service thread 1-12) JBAS016008: Starting weld service for deployment clusterbench-ee6.ear
[JBossINF] 17:19:13,865 INFO  [stdout] (pool-13-thread-1) 
[JBossINF] 17:19:13,865 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------
[JBossINF] 17:19:13,865 INFO  [stdout] (pool-13-thread-1) GMS: address=perf18/web, cluster=web, physical address=10.16.90.54:55200
[JBossINF] 17:19:13,866 INFO  [stdout] (pool-13-thread-1) -------------------------------------------------------------------
[JBossINF] 17:19:13,869 INFO  [stdout] (pool-19-thread-1) 
[JBossINF] 17:19:13,869 INFO  [stdout] (pool-19-thread-1) -------------------------------------------------------------------
[JBossINF] 17:19:13,869 INFO  [stdout] (pool-19-thread-1) GMS: address=perf18/ejb, cluster=ejb, physical address=10.16.90.54:55200
[JBossINF] 17:19:13,870 INFO  [stdout] (pool-19-thread-1) -------------------------------------------------------------------
[JBossINF] 17:19:14,195 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-12) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:19:14,195 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-11) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:19:14,268 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel
[JBossINF] 17:19:14,268 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000078: Starting JGroups Channel
[JBossINF] 17:19:14,276 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000094: Received new cluster view: [perf19/ejb|5] [perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]
[JBossINF] 17:19:14,276 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [perf19/web|5] [perf19/web, perf21/web, perf20/web, perf18/web]
[JBossINF] 17:19:14,277 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is perf18/web, physical addresses are [10.16.90.54:55200]
[JBossINF] 17:19:14,277 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-21-thread-1) ISPN000079: Cache local address is perf18/ejb, physical addresses are [10.16.90.54:55200]
[JBossINF] 17:19:14,283 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-21-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
[JBossINF] 17:19:14,283 INFO  [org.infinispan.factories.GlobalComponentRegistry] (pool-15-thread-1) ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.2.FINAL
[JBossINF] 17:19:14,284 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (pool-15-thread-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:19:14,422 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-21-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,442 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-20-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,447 INFO  [org.infinispan.jmx.CacheJmxRegistration] (pool-15-thread-1) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,503 INFO  [org.jboss.as.clustering.infinispan] (pool-21-thread-1) JBAS010281: Started remote-connector-client-mappings cache from ejb container
[JBossINF] 17:19:14,503 INFO  [org.jboss.as.clustering.infinispan] (pool-15-thread-1) JBAS010281: Started repl cache from web container
[JBossINF] 17:19:14,503 INFO  [org.jboss.as.clustering.infinispan] (pool-20-thread-1) JBAS010281: Started repl cache from ejb container
[JBossINF] 17:19:14,519 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (MSC service thread 1-2) JBAS010206: Number of cluster members: 4
[JBossINF] 17:19:14,519 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (MSC service thread 1-12) JBAS010206: Number of cluster members: 4
[JBossINF] 17:19:14,555 INFO  [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-3-thread-1) Added a new EJB receiver in cluster context ejb for node perf18. Total nodes in cluster context = 1
[JBossINF] 17:19:14,573 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:19:14,576 INFO  [org.infinispan.config.ConfigurationValidatingVisitor] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:19:14,584 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,589 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,592 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform mbean server.
[JBossINF] 17:19:14,605 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container
[JBossINF] 17:19:14,606 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container
2012/03/09 17:20:09:956 EST [DEBUG][Thread-33] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - JBossStartup, server started!
[JBossINF] 17:20:13,087 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "clusterbench-ee6.ear" was rolled back with failure message Operation cancelled
[JBossINF] 17:20:13,087 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015052: Did not receive a response to the deployment operation within the allowed timeout period [60 seconds]. Check the server configuration file and the server logs to find more about the status of the deployment.
[JBossINF] 17:20:13,095 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000029: Passivating all entries to disk
[JBossINF] 17:20:13,096 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000029: Passivating all entries to disk
[JBossINF] 17:20:13,098 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-9) ISPN000030: Passivated 0 entries in 3 milliseconds
[JBossINF] 17:20:13,098 INFO  [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-16) ISPN000030: Passivated 0 entries in 2 milliseconds
[JBossINF] 17:20:13,106 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB cache from ejb container
[JBossINF] 17:20:13,106 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010282: Stopped org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl cache from ejb container
[JBossINF] 17:20:14,587 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started //default-host//clusterbench cache from web container
[JBossINF] 17:20:14,596 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:20:14,597 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-5) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.
[JBossINF] 17:20:14,844 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-18,null) JBAS010248: New cluster view for partition web: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@1a6fa30c delta: -1, merge: false)
[JBossINF] 17:20:14,844 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.ejb] (Incoming-11,null) JBAS010248: New cluster view for partition ejb: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@17397663 delta: -1, merge: false)
[JBossINF] 17:20:14,846 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [perf21/web|6] [perf21/web, perf20/web, perf18/web]
[JBossINF] 17:20:14,849 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-11,null) ISPN000094: Received new cluster view: [perf21/ejb|6] [perf21/ejb, perf20/ejb, perf18/ejb]
[JBossINF] 17:20:14,974 INFO  [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-19,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge
[JBossINF] 17:20:14,977 WARN  [org.infinispan.commands.control.CacheViewControlCommand] (OOB-15,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=//default-host//clusterbench, type=PREPARE_VIEW, sender=perf21/web, newViewId=8, newMembers=[perf21/web, perf20/web, perf18/web], oldViewId=7, oldMembers=[perf19/web, perf21/web, perf20/web]}: java.lang.IllegalStateException: Cannot prepare new view CacheView{viewId=8, members=[perf21/web, perf20/web, perf18/web]} on cache //default-host//clusterbench, we are currently preparing view CacheView{viewId=6, members=[perf19/web, perf21/web, perf20/web, perf18/web]}
[JBossINF] 	at org.infinispan.cacheviews.CacheViewInfo.prepareView(CacheViewInfo.java:102) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:482) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] 
[JBossINF] 17:20:14,975 INFO  [org.infinispan.cacheviews.CacheViewsManagerImpl] (OOB-17,null) ISPN000170: Our last committed view (CacheView{viewId=6, members=[perf19/ejb, perf21/ejb, perf20/ejb, perf18/ejb]}) is not the same as the coordinator's last committed view (CacheView{viewId=7, members=[perf19/ejb, perf21/ejb, perf20/ejb]}). This is normal during a merge
[JBossINF] 17:20:14,990 WARN  [org.infinispan.commands.control.CacheViewControlCommand] (OOB-19,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl: Received cache view prepare request after the local node has already shut down
[JBossINF] 	at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] 
[JBossINF] 17:20:15,021 WARN  [org.infinispan.commands.control.CacheViewControlCommand] (OOB-17,null) ISPN000071: Caught exception when handling command CacheViewControlCommand{cache=org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB, type=PREPARE_VIEW, sender=perf21/ejb, newViewId=11, newMembers=[perf20/ejb, perf18/ejb, perf21/ejb], oldViewId=7, oldMembers=[perf19/ejb, perf21/ejb, perf20/ejb]}: java.lang.IllegalStateException: org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB: Received cache view prepare request after the local node has already shut down
[JBossINF] 	at org.infinispan.cacheviews.CacheViewsManagerImpl.handlePrepareView(CacheViewsManagerImpl.java:488) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.control.CacheViewControlCommand.perform(CacheViewControlCommand.java:125) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:95) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:217) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:197) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:881) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:697) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final-redhat-1.jar:3.0.6.Final-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] 
[JBossINF] 17:20:15,072 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6
[JBossINF] 17:20:15,084 ERROR [org.infinispan.cacheviews.CacheViewInfo] (OOB-17,null) ISPN000164: Rolling back to cache view 7, but last committed view is 6
[JBossINF] 17:20:15,197 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-5) Initializing Mojarra 2.1.5 (SNAPSHOT 20120217) for context '/clusterbench'
[JBossINF] 17:20:16,638 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-5) Hibernate Validator 4.2.0.Final-redhat-1
[JBossINF] 17:20:16,856 INFO  [org.jboss.web] (MSC service thread 1-5) JBAS018210: Registering web context: /clusterbench
[JBossINF] 17:21:15,196 WARN  [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-14,null) ISPN000167: Rejecting state pushed by node perf21/web for view 8, there is no state transfer in progress (we are at view 9)
[JBossINF] 17:21:15,196 WARN  [org.infinispan.statetransfer.BaseStateTransferManagerImpl] (OOB-20,null) ISPN000167: Rejecting state pushed by node perf20/web for view 8, there is no state transfer in progress (we are at view 9)
[JBossINF] 17:21:16,864 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (MSC service thread 1-13) ISPN000136: Execution error: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75) [infinispan-core-5.1.2.FINAL-redhat-1.jar:5.1.2.FINAL-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61) [jboss-as-clustering-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266) [jboss-as-clustering-web-infinispan-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445)
[JBossINF] 	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995)
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874) [jboss-msc-1.0.2.GA-redhat-1.jar:1.0.2.GA-redhat-1]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] 
[JBossINF] 17:21:16,889 ERROR [org.infinispan.transaction.TransactionCoordinator] (MSC service thread 1-13) Error while processing prepare: org.infinispan.statetransfer.StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view 10
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.signalStateTransferInProgress(StateTransferLockInterceptor.java:199)
[JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:80)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130)
[JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBossINF] 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:86)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
[JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
[JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager$ClassLoaderAwareCommandInterceptor.handleDefault(DefaultEmbeddedCacheManager.java:410)
[JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
[JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
[JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:140)
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85)
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100)
[JBossINF] 	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
[JBossINF] 	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] 
[JBossINF] 17:21:16,906 ERROR [org.infinispan.transaction.tm.DummyTransaction] (MSC service thread 1-13) ISPN000109: beforeCompletion() failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, transaction=DummyTransaction{xid=DummyXid{id=6}, status=1}, lockedKeys=null, backupKeyLocks=null, viewId=6} org.infinispan.transaction.synchronization.SyncLocalTransaction@5} org.infinispan.transaction.synchronization.SynchronizationAdapter@24: org.infinispan.CacheException: Could not prepare. 
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:70)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:223)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:234)
[JBossINF] 	at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:85)
[JBossINF] 	at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:100)
[JBossINF] 	at org.infinispan.batch.BatchContainer.resolveTransaction(BatchContainer.java:123)
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:105)
[JBossINF] 	at org.infinispan.batch.BatchContainer.endBatch(BatchContainer.java:86)
[JBossINF] 	at org.infinispan.CacheImpl.endBatch(CacheImpl.java:580)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.infinispan.AbstractDelegatingCache.endBatch(AbstractDelegatingCache.java:75)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.BatchOperation.invoke(BatchOperation.java:26)
[JBossINF] 	at org.jboss.as.clustering.infinispan.invoker.RetryingCacheInvoker.invoke(RetryingCacheInvoker.java:61)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager$ForceSynchronousCacheInvoker.invoke(DistributedCacheManager.java:522)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.invoke(DistributedCacheManager.java:490)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.batch(DistributedCacheManager.java:486)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSession(DistributedCacheManager.java:277)
[JBossINF] 	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.removeSessionLocal(DistributedCacheManager.java:266)
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.clearSessions(DistributableSessionManager.java:507) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.as.web.session.DistributableSessionManager.stop(DistributableSessionManager.java:445) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:3995) [jbossweb-7.0.10.Final-redhat-1.jar:]
[JBossINF] 	at org.jboss.as.web.deployment.WebDeploymentService.stop(WebDeploymentService.java:108) [jboss-as-web-7.1.0.Final-redhat-1.jar:7.1.0.Final-redhat-1]
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1911)
[JBossINF] 	at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1874)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
[JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
[JBossINF] Caused by: javax.transaction.xa.XAException
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:160)
[JBossINF] 	at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:122)
[JBossINF] 	at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)
[JBossINF] 	... 27 more
[JBossINF]

      Attachments

        Issue Links

          relates to

          Bug - A problem that impairs or prevents the functions of the product. AS7-4143 Failover scenarios often fail with "StateTransferInProgressException: Timed out waiting for the state transfer lock, state transfer in progress for view Y"

          • Critical - To be worked on immediately following BLOCKER issues.
          • Resolved

          Bug - A problem that impairs or prevents the functions of the product. AS7-4145 Uber large clustering WARN logging message

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              rhn-engineering-rhusar Radoslav Husar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: