-
Bug
-
Resolution: Done
-
Blocker
-
None
-
8.1.0.Beta
-
None
As usual, we have a 4 nodes cluster where we deploy clusterbench;
EAP Instances are configured to use the JGroups TCP stack:
embed-server --server-config=standalone-ha.xml if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) else /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) end-if
Fail-over is triggered by killing the EAP JVM on one node at a time;
As soon as the first node in the cluster is killed, we observe this error on the other nodes; for example if NODE 1 is killed:
2025-02-18 16:19:39,524 INFO [org.jboss.test.clusterbench.web.debug.LoggerServlet] (default task-5) CLUSTERING_LIFECYCLE_MARKER: [CLIENT_TIMESTAMP=2025-02-18 16:19:39.516] KILL
a few seconds later, we observe the "because "v" is null" error on all other nodes:
NODE 2 (logged 1338 times!):
2025-02-18 16:19:41,720 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-11,ejb,wildfly2) ISPN000136: Error executing command PrepareCommand on Cache 'clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default', writing keys []: java.lang.NullPointerException: Cannot invoke "org.infinispan.container.entries.MVCCEntry.isRemoved()" because "v" is null at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.PassivationWriterInterceptor.lambda$store$0(PassivationWriterInterceptor.java:36) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.acquireKeyFromContext(PersistenceManagerImpl.java:1404) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.entriesFromCommand(PersistenceManagerImpl.java:1384) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$toMvccEntryFlowable$73(PersistenceManagerImpl.java:1377) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.drain(FlowableConcatMap.java:266) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.onNext(FlowableConcatMap.java:159) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter$FilterSubscriber.tryOnNext(FlowableFilter.java:75) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorConditionalSubscription.slowPath(FlowableFromIterable.java:380) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:131) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.onSubscribe(FlowableConcatMap.java:142) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap.subscribeActual(FlowableConcatMap.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowablePublish.connect(FlowablePublish.java:93) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableAutoConnect.subscribeActual(FlowableAutoConnect.java:50) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableGroupBy.subscribeActual(FlowableGroupBy.java:73) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager.subscribeActual(FlowableConcatMapEager.java:54) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager$ConcatMapEagerDelayErrorSubscriber.onNext(FlowableConcatMapEager.java:134) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray$ArraySubscription.slowPath(FlowableFromArray.java:163) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray$BaseArraySubscription.request(FlowableFromArray.java:91) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager$ConcatMapEagerDelayErrorSubscriber.onSubscribe(FlowableConcatMapEager.java:108) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray.subscribeActual(FlowableFromArray.java:39) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager.subscribeActual(FlowableConcatMapEager.java:54) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribeWith(Flowable.java:16206) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.lastStage(Flowable.java:20513) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.spi.NonBlockingStore.batch(NonBlockingStore.java:538) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$batchOperation$55(PersistenceManagerImpl.java:1249) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.onNext(FlowableFlatMapSingle.java:118) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter$FilterSubscriber.tryOnNext(FlowableFilter.java:75) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorConditionalSubscription.fastPath(FlowableFromIterable.java:321) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:129) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.onSubscribe(FlowableFlatMapSingle.java:106) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle.subscribeActual(FlowableFlatMapSingle.java:53) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableLastSingle.subscribeActual(FlowableLastSingle.java:45) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.single.SingleUsing.subscribeActual(SingleUsing.java:83) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribeWith(Single.java:4905) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.toCompletionStage(Single.java:5677) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.batchOperation(PersistenceManagerImpl.java:1259) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.performBatch(PersistenceManagerImpl.java:1194) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.support.DelegatingPersistenceManager.performBatch(DelegatingPersistenceManager.java:206) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.PassivationWriterInterceptor.store(PassivationWriterInterceptor.java:35) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.commitModifications(CacheWriterInterceptor.java:148) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.afterCommit(CacheWriterInterceptor.java:158) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:85) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.visitPrepareCommand(CacheWriterInterceptor.java:132) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:78) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:198) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyModificationsAndThen(EntryWrappingInterceptor.java:929) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:893) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:193) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:121) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:144) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:130) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:231) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:208) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.BatchingInterceptor.handleDefault(BatchingInterceptor.java:74) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:211) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:107) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:108) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:111) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:79) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:48) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:154) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.TxPerCacheInboundInvocationHandler.handle(TxPerCacheInboundInvocationHandler.java:37) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:123) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:77) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1527) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1454) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1663) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1655) at org.jgroups@5.3.13.Final//org.jgroups.JChannel.up(JChannel.java:764) at org.jgroups@5.3.13.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:162) at org.jgroups@5.3.13.Final//org.jgroups.stack.Protocol.up(Protocol.java:392) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FORK.up(FORK.java:175) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:161) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:867) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:255) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1169) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:941) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:570) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:711) at org.jgroups@5.3.13.Final//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:204) at org.jgroups@5.3.13.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:288) at org.jgroups@5.3.13.Final//org.jgroups.protocols.Discovery.up(Discovery.java:316) at org.jgroups@5.3.13.Final//org.jgroups.protocols.RED.up(RED.java:123) at org.jgroups@5.3.13.Final//org.jgroups.protocols.TP.passBatchUp(TP.java:1269) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:137) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:180) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor.execute(ContextualExecutor.java:31) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:220) at java.base/java.lang.Thread.run(Thread.java:840)
NODE 3 (logged 716 times!):
2025-02-18 16:19:41,717 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-6,null,wildfly3) ISPN000136: Error executing command PrepareCommand on Cache 'clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default', writing keys []: java.lang.NullPointerException: Cannot invoke "org.infinispan.container.entries.MVCCEntry.isRemoved()" because "v" is null at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.PassivationWriterInterceptor.lambda$store$0(PassivationWriterInterceptor.java:36) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.acquireKeyFromContext(PersistenceManagerImpl.java:1404) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.entriesFromCommand(PersistenceManagerImpl.java:1384) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$toMvccEntryFlowable$73(PersistenceManagerImpl.java:1377) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.drain(FlowableConcatMap.java:266) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.onNext(FlowableConcatMap.java:159) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter$FilterSubscriber.tryOnNext(FlowableFilter.java:75) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorConditionalSubscription.slowPath(FlowableFromIterable.java:380) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:131) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.onSubscribe(FlowableConcatMap.java:142) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap.subscribeActual(FlowableConcatMap.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowablePublish.connect(FlowablePublish.java:93) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableAutoConnect.subscribeActual(FlowableAutoConnect.java:50) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableGroupBy.subscribeActual(FlowableGroupBy.java:73) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager.subscribeActual(FlowableConcatMapEager.java:54) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager$ConcatMapEagerDelayErrorSubscriber.onNext(FlowableConcatMapEager.java:134) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray$ArraySubscription.slowPath(FlowableFromArray.java:163) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray$BaseArraySubscription.request(FlowableFromArray.java:91) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager$ConcatMapEagerDelayErrorSubscriber.onSubscribe(FlowableConcatMapEager.java:108) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromArray.subscribeActual(FlowableFromArray.java:39) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMapEager.subscribeActual(FlowableConcatMapEager.java:54) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribeWith(Flowable.java:16206) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.lastStage(Flowable.java:20513) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.spi.NonBlockingStore.batch(NonBlockingStore.java:538) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$batchOperation$55(PersistenceManagerImpl.java:1249) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.onNext(FlowableFlatMapSingle.java:118) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter$FilterSubscriber.tryOnNext(FlowableFilter.java:75) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorConditionalSubscription.fastPath(FlowableFromIterable.java:321) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:129) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.onSubscribe(FlowableFlatMapSingle.java:106) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:66) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle.subscribeActual(FlowableFlatMapSingle.java:53) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16149) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16095) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.flowable.FlowableLastSingle.subscribeActual(FlowableLastSingle.java:45) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.internal.operators.single.SingleUsing.subscribeActual(SingleUsing.java:83) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.subscribeWith(Single.java:4905) at io.reactivex.rxjava3.rxjava//io.reactivex.rxjava3.core.Single.toCompletionStage(Single.java:5677) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.batchOperation(PersistenceManagerImpl.java:1259) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.manager.PersistenceManagerImpl.performBatch(PersistenceManagerImpl.java:1194) at org.infinispan.core@15.0.11.Final//org.infinispan.persistence.support.DelegatingPersistenceManager.performBatch(DelegatingPersistenceManager.java:206) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.PassivationWriterInterceptor.store(PassivationWriterInterceptor.java:35) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.commitModifications(CacheWriterInterceptor.java:148) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.afterCommit(CacheWriterInterceptor.java:158) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:85) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.CacheWriterInterceptor.visitPrepareCommand(CacheWriterInterceptor.java:132) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:78) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:198) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyModificationsAndThen(EntryWrappingInterceptor.java:929) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:893) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:193) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:102) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:121) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:144) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:130) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:231) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:208) at org.infinispan.core@15.0.11.Final//org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.BatchingInterceptor.handleDefault(BatchingInterceptor.java:74) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:137) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:196) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50) at org.infinispan.core@15.0.11.Final//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:211) at org.infinispan.core@15.0.11.Final//org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:107) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:108) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:111) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:79) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:48) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:154) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.TxPerCacheInboundInvocationHandler.handle(TxPerCacheInboundInvocationHandler.java:37) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:123) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:77) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1527) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1454) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1663) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1655) at org.jgroups@5.3.13.Final//org.jgroups.JChannel.up(JChannel.java:764) at org.jgroups@5.3.13.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:162) at org.jgroups@5.3.13.Final//org.jgroups.stack.Protocol.up(Protocol.java:392) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FORK.up(FORK.java:175) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:161) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:867) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:255) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1169) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:941) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:570) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:711) at org.jgroups@5.3.13.Final//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:204) at org.jgroups@5.3.13.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:288) at org.jgroups@5.3.13.Final//org.jgroups.protocols.Discovery.up(Discovery.java:316) at org.jgroups@5.3.13.Final//org.jgroups.protocols.RED.up(RED.java:123) at org.jgroups@5.3.13.Final//org.jgroups.protocols.TP.passBatchUp(TP.java:1269) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:137) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:180) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor.execute(ContextualExecutor.java:31) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:220) at java.base/java.lang.Thread.run(Thread.java:840)
NODE 4:
2025-02-18 16:19:41,733 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-26,ejb,wildfly4) ISPN000136: Error executing command PrepareCommand on Cache 'clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default', writing keys [InfinispanBeanMetaDataKey(UUIDSessionID [a6a4fd92-b601-437d-9e64-ecd063a15090]), InfinispanBeanGroupKey(UUIDSessionID [a6a4fd92-b601-437d-9e64-ecd063a15090])]: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from wildfly3, see cause for remote stack trace at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:26) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.impl.MapResponseCollector.addException(MapResponseCollector.java:65) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.impl.MapResponseCollector$IgnoreLeavers.addException(MapResponseCollector.java:103) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.ValidResponseCollector.addResponse(ValidResponseCollector.java:29) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onResponse(MultiTargetRequest.java:98) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1556) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1457) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.lambda$up$0(JGroupsTransport.java:1663) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1655) at org.jgroups@5.3.13.Final//org.jgroups.JChannel.up(JChannel.java:764) at org.jgroups@5.3.13.Final//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:162) at org.jgroups@5.3.13.Final//org.jgroups.stack.Protocol.up(Protocol.java:392) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FORK.up(FORK.java:175) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FRAG2.up(FRAG2.java:161) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FlowControl.up(FlowControl.java:319) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.GMS.up(GMS.java:867) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:255) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1169) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:941) at org.jgroups@5.3.13.Final//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:570) at org.jgroups@5.3.13.Final//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:711) at org.jgroups@5.3.13.Final//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193) at org.jgroups@5.3.13.Final//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:204) at org.jgroups@5.3.13.Final//org.jgroups.protocols.MERGE3.up(MERGE3.java:288) at org.jgroups@5.3.13.Final//org.jgroups.protocols.Discovery.up(Discovery.java:316) at org.jgroups@5.3.13.Final//org.jgroups.protocols.RED.up(RED.java:123) at org.jgroups@5.3.13.Final//org.jgroups.protocols.TP.passBatchUp(TP.java:1269) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.passBatchUp(SubmitToThreadPool.java:137) at org.jgroups@5.3.13.Final//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:133) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:180) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.ContextualExecutor.execute(ContextualExecutor.java:31) at org.wildfly.clustering.context@5.0.4.Final//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:220) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException: Cannot invoke "org.infinispan.container.entries.MVCCEntry.isRemoved()" because "v" is null at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.exts.ThrowableExternalizer.newThrowableInstance(ThrowableExternalizer.java:286) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readGenericThrowable(ThrowableExternalizer.java:264) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:239) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:44) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan.core@15.0.11.Final//org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.wildfly.clustering.cache.infinispan.embedded@5.0.4.Final//org.wildfly.clustering.cache.infinispan.embedded.marshall.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:50) at org.infinispan.core@15.0.11.Final//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1548) ... 32 more
Complete logs ans server config in eap-8.x-clustering-ejb-ejbservlet-jvmkill-dist#264.zip
Client fail rate isn't affected but the logs are literally flooded;
We marked this as blocker bacsue we didn't observe the issue in EAP 8.0.6.GA-CR2;
- is duplicated by
-
-
- Verified
-
- is incorporated by
-
JBEAP-29470 Upgrade Infinispan to 15.0.14.Final
-
- Verified
-