Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-5573

Node shutting down can throw IllegalLifecycleStateException wrapped multiple times

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 8.0.0.Beta1
    • 8.0.0.Alpha2
    • Core
    • None

      When working on some new work I have a stress test that constantly stops and starts a cache to test the resiliency of my code.

      I found that every once in a while if I have something that takes a little longer to serialize it could throw back a non SuspectException to the originator. It looks like the underlying issue is that IllegalLifecycleStateException is wrapped by multiple exceptions and thus it comes back as a simple CacheException instead which treats the exception properly.

      15:31:48,053 TRACE (OOB-2,main-NodeD-61136:) [ClusterStreamManagerImpl] Encounted exception for cf133b23-9bbf-49ee-823d-f2f650f39d9c from main-NodeH-22716
      java.util.concurrent.CompletionException: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeH-22716, see cause for remote stack trace
      	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
      	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
      	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:824)
      	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
      	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
      	at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.futureDone(SingleResponseFuture.java:27)
      	at org.jgroups.blocks.Request.checkCompletion(Request.java:169)
      	at org.jgroups.blocks.UnicastRequest.receiveResponse(UnicastRequest.java:83)
      	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:398)
      	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
      	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
      	at org.jgroups.JChannel.up(JChannel.java:739)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
      	at org.jgroups.protocols.RSVP.up(RSVP.java:201)
      	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
      	at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
      	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
      	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
      	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:291)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
      	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeH-22716, see cause for remote stack trace
      	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:747)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$76(JGroupsTransport.java:586)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$5/1861463967.apply(Unknown Source)
      	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
      	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      	... 26 more
      Caused by: org.infinispan.commons.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:127)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:119)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:565)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotelyAsync(RpcManagerImpl.java:180)
      	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222)
      	at org.infinispan.stream.impl.LocalStreamManagerImpl.streamOperationRehashAware(LocalStreamManagerImpl.java:240)
      	at org.infinispan.stream.impl.StreamRequestCommand.perform(StreamRequestCommand.java:93)
      	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:85)
      	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:34)
      	... 3 more
      Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:238)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:252)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:116)
      	... 10 more
      Caused by: org.infinispan.IllegalLifecycleStateException: Cache marshaller has been stopped
      	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.getObjectWriter(JBossMarshaller.java:146)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:132)
      	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      	at org.infinispan.commons.marshall.MarshallUtil.marshallMap(MarshallUtil.java:36)
      	at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:60)
      	at org.infinispan.marshall.exts.MapExternalizer.writeObject(MapExternalizer.java:28)
      	at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
      	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:58)
      	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128)
      	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112)
      	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:68)
      	at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:442)
      	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
      	at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
      	at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
      	at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:72)
      	at org.infinispan.marshall.core.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:77)
      	at org.infinispan.commons.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:41)
      	at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:85)
      	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:23)
      	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:236)
      	... 12 more
      Caused by: an exception which occurred:
      	in object java.lang.Integer@20d5
      	in object java.util.HashMap@0
      	in object org.infinispan.stream.impl.StreamResponseCommand@66d77895
      

              wburns@redhat.com Will Burns
              wburns@redhat.com Will Burns
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: