SingleKeyBackupWriteCommands run out of memory

SingleKeyBackupWriteCommands run out of memory if sender does send out single SingleKeyBackupWriteCommand for some reason(Exception or deadlock during org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups

long sequence = triangleOrderManager.next(segmentId, topologyId);

//Sequence is increased but the cmd has no chance to send out for there is exception/deadlock later.
BackupWriteCommand backupCommand = backupBuilder.build(commandsFactory, command, entry.getValue());
backupCommand.setSequence(sequence);
backupCommand.setSegmentId(segmentId);
if (trace)

{ log.tracef("Command %s got sequence %s for segment %s", command.getCommandInvocationId(), segmentId, sequence); }

rpcManager.sendToMany(backups, backupCommand, DeliverOrder.NONE);

16-Oct-2020 01:09:38.403 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl.rethrowAsCacheException] (dp-27-thread-1) ISPN000073: Unexpected error while re
plicating
java.lang.RuntimeException: What's happen if Externalizer has exception?
at com.ericsson.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:397)
at com.example.container.sip.fsm.InviteSessionFsm$InviteSessionExternalizer.writeObject(InviteSessionFsm.java:385)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:930)
at com.example.ssa.sip.UA$UAExternalizer.writeObject(UA.java:919)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)

at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:780)
at com.example.container.sip.session.dialog.DialogFragmentData$DialogFragmentDataExternalizer.writeObject(DialogFragmentData.java:761)
at com.example.sessiondataservices.serialization.AdvancedExternalizerBridge.writeObject(AdvancedExternalizerBridge.java:42)
at org.infinispan.marshall.core.impl.ExternalExternalizers$ForeignAdvancedExternalizer.writeObject(ExternalExternalizers.java:86)
at org.infinispan.marshall.core.GlobalMarshaller.writeExternal(GlobalMarshaller.java:670)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:375)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26)
at org.infinispan.commands.triangle.SingleKeyBackupWriteCommand.writeTo(SingleKeyBackupWriteCommand.java:108)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:70)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:122)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:118)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:69)
at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656)
at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371)
at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181)
at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174)
at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.marshallRequest(JGroupsTransport.java:1075)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendCommand(JGroupsTransport.java:1272)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendToMany(JGroupsTransport.java:277)
at org.infinispan.remoting.rpc.RpcManagerImpl.sendToMany(RpcManagerImpl.java:396)
at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.sendToBackups(TriangleDistributionInterceptor.java:490)
at org.infinispan.interceptors.distribution.TriangleDistributionInterceptor.lambda$localPrimaryOwnerWrite$4(TriangleDistributionInterceptor.java:446)

The receiver won't handle the subsquent SingleKeyBackupWriteCommands for current SingleKeyBackupWriteCommand is missing. Then the  subsquent SingleKeyBackupWriteCommands will be putted into blockedTasks.offer(runnable); by BlockingTaskAwareExecutorServiceImpl instead of execute it.