Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-12435

SingleKeyBackupWriteCommands run out of memory

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 12.1.0.Final
    • 10.1.8.Final
    • Cloud
    • None

      SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups

      long sequence = triangleOrderManager.next(segmentId, topologyId);

      //Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.
      BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
      backupCommand.setSequence(sequence);
      backupCommand.setSegmentId(segmentId);
      if (trace)

      { log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId, sequence); }

      rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);

      16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
      plicating
      java.lang.RuntimeException: What's happen if Externalizer has exception?
      at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397)
      at com.example.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
      at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
      at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
      at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
      at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
      at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
      at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
      at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
      at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
      at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
      at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
      at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
      at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
      at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
      at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)

      at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
      at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
      at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
      at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
      at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
      at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
      at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
      at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
      at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
      at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
      at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
      at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
      at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
      at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
      at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
      at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
      at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
      at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
      at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490)
      at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)

      The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the  subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.

       

              dberinde@redhat.com Dan Berindei (Inactive)
              eyanpenmac mac peng (Inactive)
              Archiver:
              rhn-support-adongare Amol Dongare

                Created:
                Updated:
                Resolved:
                Archived: