-
Bug
-
Resolution: Done
-
Major
-
14.0.21.Final
-
None
I have the following configuration on the server:
<cache-container name="default" statistics="true"> <jmx enabled="true"/> <metrics accurate-size="true"/> <transport cluster="${infinispan.cluster.name:cluster}" /> <replicated-cache mode="SYNC" statistics="true" name="myConf"> <memory storage="OFF_HEAP" max-count="10" when-full="REMOVE"/> <persistence passivation="true"> <file-store shared="false" preload="false" purge="true"/> </persistence> </replicated-cache> </cache-container>
I have a simple application which puts 600k entries into the cache in parallel threads, i.e. 6 threads each is putting it's portion of entries, each puts 100k entries into the cache over Hot Rod client. In 4 runs from 5, I am starting to get the following exception on the server and on the client:
Server exception:
2023-11-22 11:58:34,387 ERROR (non-blocking-thread--p2-t21) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand on Cache 'myConf', writing keys [WrappedByteArray[\J09\k\e\y\5\0\2\5\7\6 (11 bytes)]] org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\5\0\2\5\7\6 (11 bytes)] and requestor CommandInvocation:fedora-20723:15519. Lock is held by CommandInvocation:fedora-20723:15515 at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:299) at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:229) at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:440) at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:416) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.lang.Thread.run(Thread.java:833)
Client exception:
WARN: ISPN004005: Error received from the server: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\1\0\0\5\8\0 (11 bytes)] and requestor CommandInvocation:fedora-34014:3514. Lock is held by CommandInvocation:fedora-34014:3512 [WARNING] org.infinispan.client.hotrod.exceptions.HotRodClientException: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\1\0\0\5\8\0 (11 bytes)] and requestor CommandInvocation:fedora-34014:3514. Lock is held by CommandInvocation:fedora-34014:3512 at java.util.concurrent.CompletableFuture.reportGet (CompletableFuture.java:396) at java.util.concurrent.CompletableFuture.get (CompletableFuture.java:2096) at org.infinispan.client.hotrod.impl.Util.await (Util.java:52) at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put (RemoteCacheSupport.java:196) at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put (RemoteCacheSupport.java:186) at test.Main2$Loader.run (Main2.java:50)
The issue appears for both OFF_HEAP and HEAP memory config. The issue doesn't appear if `memory` tag is removed.
The reproducer java file is attached. Also attaching the thread dump for one of the similar runs.
