Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7558

IllegalLifecycleStateException due to Cache marshaller has been stopped

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Minor
    • 9.1.0.Final
    • 8.2.6.Final
    • Core
    • None

    Description

      We saw this error in failover scenarios:

      14:28:41,783 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 85) WFLYCLINF0003: Stopped client-mappings cache from ejb container
      [JBossINF] �14:28:41,783 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 77) WFLYCLINF0003: Stopped routing cache from web container
      [JBossINF] 14:28:41,790 INFO  [org.infinispan.CLUSTER] (remote-thread--p8-t3) [Context=client-mappings][Scope=perf20]ISPN100003: Finished local rebalance
      [JBossINF] 14:28:41,791 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000080: Disconnecting JGroups channel ejb
      [JBossINF] 14:28:41,791 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) ISPN000082: Stopping the RpcDispatcher for channel ejb
      [JBossINF] 14:28:41,792 INFO  [org.infinispan.CLUSTER] (remote-thread--p7-t6) [Context=routing][Scope=perf20]ISPN100003: Finished local rebalance
      [JBossINF] 14:28:41,797 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel web
      [JBossINF] 14:28:41,799 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
      [JBossINF] 14:28:41,818 ERROR [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (thread-19) ISPN000065: Exception while marshalling object: CacheNotFoundResponse: org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped
      [JBossINF] 	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getExternalizerTable(JBossMarshaller.java:155)
      [JBossINF] 	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:144)
      [JBossINF] 	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132)
      [JBossINF] 	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      [JBossINF] 	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      [JBossINF] 	at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)
      [JBossINF] 	at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)
      [JBossINF] 	at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
      [JBossINF] 	at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:248)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.lambda$executeCommandFromLocalCluster$1(CommandAwareRpcDispatcher.java:199)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:121)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:75)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:200)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:169)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
      [JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:738)
      [JBossINF] 	at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
      [JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:380)
      [JBossINF] 	at org.jgroups.protocols.FORK.up(FORK.java:114)
      [JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:374)
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
      [JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1037)
      [JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
      [JBossINF] 	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:442)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:968)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:848)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
      [JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
      [JBossINF] 	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
      [JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
      [JBossINF] 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
      [JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:296)
      [JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
      [JBossINF] 	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1872)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: an exception which occurred:
      [JBossINF] 	in object org.infinispan.IllegalLifecycleStateException@244b87df
      [JBossINF] 		-> toString = org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped
      [JBossINF] 
      [JBossINF] �14:28:41,912 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0009: Undeployed "clusterbench-ee7.ear" (runtime-name: "clusterbench-ee7.ear")
      

      Error occurs on server in three async scenarios during server stopping.

      Client is not affected.

      Scenarios affected:
      ejb-ejbremote-shutdown-repl-async (2 occurences)
      ejb-ejbremote-undeploy-dist-async (1 occurence)
      ejb-ejbservlet-undeploy-repl-sync (2 occurences)
      http-granular-shutdown-dist-sync (1 occurence)
      http-session-shutdown-repl-async (1 occurence)
      http-session-shutdown-concurrent-async (1 occurence)

      Link to server log:
      http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbremote-undeploy-dist-async/11/console-perf18/

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rhn-engineering-rhusar Radoslav Husar
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: