-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
8.1.0.GA-CR3
-
None
-
False
-
-
False
-
-
-
-
-
-
-
After https://issues.redhat.com/browse/JBEAP-29906 has been solved, we still observe some ISPN000136 / ISPN000299 errors in the following tests scenarios:
- "EJB + SERVLET" HTTP distributed session tests with MOD_JK load-balancer and TCP JGroups stack (cache is set to distributed, as by default) where fail-over is introduced by EAP shut-down
- "SERVLET" HTTP distributed session tests with HAPROXY (with dedicated affinity-cookie) load-balancer and TCP JGroups stack (cache is set to distributed, as by default) where fail-over is introduced by EAP JVM-Kill
- "SERVLET" HTTP distributed session tests with MOD_JK load-balancer and TCP JGroups stack (cache is set to distributed, as by default) where fail-over is introduced by un-deploying from EAP the distributed application
- "SERVLET" HTTP distributed session tests with MOD_JK load-balancer and TCP JGroups stack (cache is set to distributed, as by default with cache-container's "granularity=ATTRIBUTE") where fail-over is introduced by EAP shut-down
We are providing all the details in the comments below; here we just print the error observed in the most affected scenario (the one with "granularity=ATTRIBUTE"); in this case, the issue is observed after the first node of the cluster (the first that was originally started and probably the coordinator) is shut-down and then re-started after the initial phase where all 4 nodes in the cluster are up and running and have been serving requests for several minutes without any issue:
2025-06-04 05:14:28,884 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (thread-36,ejb,wildfly1) ISPN000136: Error executing command LockControlCommand on Cache 'clusterbench-ee10.ear.clusterbench-ee10-web.war', writing keys []: org.infinispan.commons.TimeoutException: ISPN000299: Unable to acquire lock after 0 milliseconds for key SessionMetaDataKey(oRGryejk-Zs_PwaWc0OnHrM-YOCMrCwV3WJi1I5Z) and requestor GlobalTransaction{id=54140, addr=wildfly3, remote=true, xid=null, internalId=-1}. Lock is held by GlobalTransaction{id=54130, addr=wildfly3, remote=true, xid=null, internalId=-1}
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:299)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:229)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:554)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.toInvocationStage(InfinispanLock.java:481)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:259)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.toInvocationStage(DefaultLockManager.java:283)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAllAndRecord(AbstractLockingInterceptor.java:293)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockAllKeys(AbstractTxLockingInterceptor.java:157)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAllOrRegisterBackupLock(AbstractTxLockingInterceptor.java:109)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.PessimisticLockingInterceptor.localLockCommandWork(PessimisticLockingInterceptor.java:211)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.PessimisticLockingInterceptor.lambda$new$0(PessimisticLockingInterceptor.java:44)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:85)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitLockControlCommand(PessimisticLockingInterceptor.java:193)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:102)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:187)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.impl.TxInterceptor.visitLockControlCommand(TxInterceptor.java:224)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:102)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.asyncInvokeNext(BaseAsyncInterceptor.java:231)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:46)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:189)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.statetransfer.StateTransferInterceptor.visitLockControlCommand(StateTransferInterceptor.java:90)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:102)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:58)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:165)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:102)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:127)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:60)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.impl.BatchingInterceptor.handleDefault(BatchingInterceptor.java:74)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.DDAsyncInterceptor.visitLockControlCommand(DDAsyncInterceptor.java:165)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.acceptVisitor(LockControlCommand.java:102)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:50)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:211)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.commands.control.LockControlCommand.invokeAsync(LockControlCommand.java:112)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:108)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:111)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:79)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:48)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.handleRunnable(BasePerCacheInboundInvocationHandler.java:154)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.TxPerCacheInboundInvocationHandler.handle(TxPerCacheInboundInvocationHandler.java:37)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleCacheRpcCommand(GlobalInboundInvocationHandler.java:123)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:77)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1527)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1454)
at org.infinispan.core@15.0.14.Final-redhat-00002//org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1649)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.JChannel.up(JChannel.java:748)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:132)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:360)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FORK.up(FORK.java:146)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:253)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:261)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:850)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:235)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1144)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:880)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:862)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:474)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:669)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:190)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:296)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:360)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.protocols.TP.passMessageUp(TP.java:1230)
at org.jgroups@5.3.15.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:95)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.wildfly.clustering.context@5.0.10.Final-redhat-00001//org.wildfly.clustering.context.ContextualExecutor$1.execute(ContextualExecutor.java:180)
at org.wildfly.clustering.context@5.0.10.Final-redhat-00001//org.wildfly.clustering.context.ContextualExecutor.execute(ContextualExecutor.java:31)
at org.wildfly.clustering.context@5.0.10.Final-redhat-00001//org.wildfly.clustering.context.Contextualizer$2$1.run(Contextualizer.java:220)
at java.base/java.lang.Thread.run(Thread.java:840)
- is related to
-
-
- Verified
-