Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-5330

Various issues with stopping a cache/cachemanager

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 7.1.1.Final
    • Fix Version/s: 8.0.0.Beta1
    • Component/s: None
    • Labels:
      None

      Description

      While running stress test that constantly starts and stops a new cache manager I found several issues where a server would respond with incorrect values or exceptions.

      1. The Marshaller objectTable can be nulled out causing a NPE.
        Caused by: java.lang.NullPointerException
        	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
        	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:84)
        	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:1)
        	at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
        	at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
        	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        	at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
        	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:153)
        	at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:1)
        	at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:434)
        	at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:226)
        	at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
        	at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
        	at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
        	at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
        	at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
        	at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
        	at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
        	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:245)
        	... 22 more
        
      2. A command can be ran while a cache is STOPPING causing possibly invalid values to be returned. We need to most likely tweak the DistributionResponseGenerator for various commands to drop results that are null for GetKeyValueCommand for example (instead return UnsureResponse)
      3. If a cache has gone to TERMINATING state an exception is thrown back to the caller which is majority of cases not handled
        org.infinispan.remoting.RemoteException: ISPN000217: Received exception from main-NodeIJ-43170, see cause for remote stack trace
        	at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:44)
        	at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:69)
        	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:656)
        	at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:346)
        	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.retrieveFromRemoteSources(BaseDistributionInterceptor.java:223)
        	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitGetAllCommand(NonTxDistributionInterceptor.java:142)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetAllCommand(EntryWrappingInterceptor.java:150)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitGetAllCommand(NonTransactionalLockingInterceptor.java:50)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
        	at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.statetransfer.StateTransferInterceptor.visitGetAllCommand(StateTransferInterceptor.java:177)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetAllCommand(CacheMgmtInterceptor.java:125)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
        	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
        	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
        	at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
        	at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:448)
        	at org.infinispan.cache.impl.CacheImpl.getAll(CacheImpl.java:442)
        	at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:117)
        	at org.infinispan.commands.GetAllCommandStressTest$1.call(GetAllCommandStressTest.java:1)
        	at org.infinispan.test.AbstractInfinispanTest$LoggingCallable.call(AbstractInfinispanTest.java:422)
        	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        	at java.lang.Thread.run(Thread.java:745)
        Caused by: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'org.infinispan.commands.GetAllCommandStressTest' is in 'STOPPING' state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
        	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:91)
        	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
        	at org.infinispan.commands.AbstractVisitor.visitGetAllCommand(AbstractVisitor.java:96)
        	at org.infinispan.commands.read.GetAllCommand.acceptVisitor(GetAllCommand.java:59)
        	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
        	at org.infinispan.commands.remote.ClusteredGetManyCommand.perform(ClusteredGetManyCommand.java:83)
        	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokePerform(BasePerCacheInboundInvocationHandler.java:84)
        	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:31)
        	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:124)
        	at org.infinispan.remoting.inboundhandler.NonTotalOrderPerCacheInboundInvocationHandler.handle(NonTotalOrderPerCacheInboundInvocationHandler.java:53)
        	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:125)
        	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:74)
        	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:326)
        	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
        	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
        	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
        	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
        	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
        	at org.jgroups.JChannel.up(JChannel.java:739)
        	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
        	at org.jgroups.protocols.RSVP.up(RSVP.java:201)
        	at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
        	at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
        	at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
        	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
        	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
        	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
        	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
        	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
        	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
        	at org.jgroups.protocols.Discovery.up(Discovery.java:291)
        	at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
        	at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
        	... 3 more
        

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  william.burns Will Burns
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: