Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-5494

Node replies with NullPointerException during shutdown

    XMLWordPrintable

    Details

      Description

      When a node is shutting down, the CacheMarshaller is stopped before the JGroupsTransport. Commands that have been received after JBM was stopped cannot be deserialized, and CommandAwareRpcDispatcher will reply with a CacheException(NullPointerException:

      22:17:26,119 ERROR (HotRodServerWorker-404-1:) [InvocationContextInterceptor] ISPN000136: Execution error
      org.infinispan.remoting.RemoteException: ISPN000217: Received exception from DistTopologyChangeUnderLoadTest-NodeB-4216, see cause for remote stack trace
          at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:46)
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:436)
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:240)
          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:561)
          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:287)
          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:339)
          at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:105)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:388)
          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:453)
          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:195)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:88)
          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:40)
          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:55)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)
          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:324)
          at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:256)
          at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:115)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)
          at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
          at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)
          at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)
          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
          at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)
          at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1097)
          at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1089)
          at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:522)
          at org.infinispan.server.hotrod.CacheDecodeContext.put(CacheDecodeContext.scala:216)
          at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$hotrod$HotRodDecoder$$decodeValue(HotRodDecoder.scala:132)
          at org.infinispan.server.hotrod.HotRodDecoder$$anonfun$decode$1.apply$mcV$sp(HotRodDecoder.scala:50)
          at org.infinispan.server.hotrod.HotRodDecoder.wrapSecurity(HotRodDecoder.scala:208)
          at org.infinispan.server.hotrod.HotRodDecoder.decode(HotRodDecoder.scala:45)
          at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:370)
          at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)
          at org.infinispan.server.hotrod.HotRodDecoder.org$infinispan$server$core$transport$StatsChannelHandler$$super$channelRead(HotRodDecoder.scala:31)
          at org.infinispan.server.core.transport.StatsChannelHandler$class.channelRead(StatsChannelHandler.scala:32)
          at org.infinispan.server.hotrod.HotRodDecoder.channelRead(HotRodDecoder.scala:31)
          at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
          at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
          at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
          at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
          at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
          at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
          at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
          at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
          at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
          at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
          at java.lang.Thread.run(Thread.java:745)
      Caused by: org.infinispan.commons.CacheException: Problems invoking command.
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:312)
          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:250)
          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:675)
          at org.jgroups.JChannel.up(JChannel.java:739)
          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1029)
          at org.jgroups.protocols.RSVP.up(RSVP.java:201)
          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
          at org.jgroups.protocols.FlowControl.up(FlowControl.java:394)
          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)
          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1042)
          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)
          at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1064)
          at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:779)
          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:426)
          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:652)
          at org.jgroups.protocols.Discovery.up(Discovery.java:291)
          at org.jgroups.protocols.TP.passMessageUp(TP.java:1577)
          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1796)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          ... 1 more
      Caused by: java.lang.NullPointerException
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:101)
          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:154)
          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:65)
          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:436)
          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:227)
          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:148)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:354)
          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135)
          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)
          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)
          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)
          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:297)
          ... 22 more
      

      This is causing random test failures, e.g. DistTopologyChangeUnderLoadTest.testPutsSucceedWhileTopologyChanges:
      http://ci.infinispan.org/viewLog.html?buildId=26516&tab=buildResultsDiv&buildTypeId=Infinispan_MasterHotspotJdk7trac

      Because it's a cache-scoped component, CacheMarshaller cannot be stopped after JGroupsTransport, but the CacheException should be replaced with a CacheNotFoundResponse when the node is shutting down.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dberinde@redhat.com Dan Berindei
              Reporter:
              dberinde@redhat.com Dan Berindei
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: