Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-8269

Exception while marshalling object: Cache manager is shutting down

    XMLWordPrintable

Details

    Description

      In failover tests we saw an error on server. It occured during server stopping and application undeployment.

      Server log stacktrace:

      12:54:10,382 ERROR [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (thread-19,ee,perf18) ISPN000065: Exception while marshalling object: ExceptionResponse: java.io.IOException: ISPN000239: Cache manager is shutting down, so type write externalizer for type=java.io.IOException cannot be resolved
      [JBossINF] 	at org.infinispan.marshall.core.ExternalizerTable.getObjectWriter(ExternalizerTable.java:197)
      [JBossINF] 	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132)
      [JBossINF] 	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      [JBossINF] 	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      [JBossINF] 	at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)
      [JBossINF] 	at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToBuffer(AbstractJBossMarshaller.java:80)
      [JBossINF] 	at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.reply(CommandAwareRpcDispatcher.java:248)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:183)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:455)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:406)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:357)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:245)
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:664)
      [JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:738)
      [JBossINF] 	at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:120)
      [JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:380)
      [JBossINF] 	at org.jgroups.protocols.FORK.up(FORK.java:114)
      [JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:374)
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
      [JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1037)
      [JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
      [JBossINF] 	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:442)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.deliver(NAKACK2.java:968)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:848)
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:611)
      [JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:155)
      [JBossINF] 	at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:200)
      [JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:325)
      [JBossINF] 	at org.jgroups.protocols.MERGE3.up(MERGE3.java:292)
      [JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:296)
      [JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1657)
      [JBossINF] 	at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1873)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] Caused by: an exception which occurred:
      [JBossINF] 	in object java.io.IOException@314c010
      [JBossINF] 		-> toString = java.io.IOException: ISPN000239: Cache manager is shutting down, so type write externalizer for type=org.infinispan.remoting.responses.ExceptionResponse cannot be resolved
      [JBossINF] 
      [JBossINF] 12:54:10,585 INFO  [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 79) ISPN000030: Passivated 4000 entries in 364 milliseconds
      12:54:10,594 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 79) WFLYCLINF0003: Stopped clusterbench-ee7.ear/clusterbench-ee7-ejb.jar cache from ejb container
      

      Occurs after

      12:54:10,371 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel web
      

      There are NoSuchEJBException error messages on client at the same time in scenario eap-7x-failover-ejb-ejbremote-shutdown-repl-async:

      12:54:10,468 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-1) Retrying invocation which failed on node perf18 with exception:
      javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee7,modulename=clusterbench-ee7-ejb,distinctname=,beanname=RemoteStatefulSBImpl,viewclassname=org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSB]
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:395)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:507)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:434)
      	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:789)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      and WARN messages in scenario eap-7x-failover-ejb-ejbservlet-undeploy-repl-sync

      14:50:29:275 EST [WARN ][Runner - 159] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
              org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
      	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
      	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
      	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
      	at java.lang.Thread.run(Thread.java:745)
      

      Scenarios affected:
      eap-7x-failover-ejb-ejbremote-shutdown-repl-async (1 occurence)
      eap-7x-failover-ejb-ejbservlet-undeploy-repl-sync (1 occurence)
      eap-7x-failover-jvmkill-ha-ss (1 occurence)

      Link to server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbremote-shutdown-repl-async/12/console-perf18/

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              dcihak@redhat.com Daniel Cihak
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: