Details
Major Description
When I run the client test suite on a single CPU with taskset -c 1, it often hangs in one test or another. The main thread is waiting for NonRecursiveEventLoopGroup to shut down
"testng-ReplicationHitsTest" #18 prio=5 os_prio=0 cpu=14562.91ms elapsed=631.88s tid=0x00007fd5451ac000 nid=0x324051 in Object.wait() [0x00007fd50448c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(java.base@11.0.10/Native Method) - waiting on <0x00000000c414a448> (a io.netty.util.concurrent.DefaultPromise) at java.lang.Object.wait(java.base@11.0.10/Object.java:328) at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:253) - waiting to re-lock in wait() <0x00000000c414a448> (a io.netty.util.concurrent.DefaultPromise) at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:35) at org.infinispan.server.core.transport.NonRecursiveEventLoopGroup.shutdownGracefullyAndWait(NonRecursiveEventLoopGroup.java:82)
And another thread seems to be in an infinite loop, retrying the same task over and over again
"non-blocking-thread-ReplicationHitsTest-NodeC-p1011-t5" #4906 daemon prio=5 os_prio=0 cpu=30662.18ms elapsed=308.85s tid=0x00007fd3ec01e800 nid=0x325b56 runnable [0x00007fd41f3f3000] java.lang.Thread.State: RUNNABLE at java.lang.Throwable.fillInStackTrace(java.base@11.0.10/Native Method) at java.lang.Throwable.fillInStackTrace(java.base@11.0.10/Throwable.java:787) - locked <0x00000000ec1ffd20> (a java.util.concurrent.RejectedExecutionException) at java.lang.Throwable.<init>(java.base@11.0.10/Throwable.java:270) at java.lang.Exception.<init>(java.base@11.0.10/Exception.java:66) at java.lang.RuntimeException.<init>(java.base@11.0.10/RuntimeException.java:62) at java.util.concurrent.RejectedExecutionException.<init>(java.base@11.0.10/RejectedExecutionException.java:64) at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) at org.infinispan.server.core.transport.NonRecursiveEventLoopGroup.execute(NonRecursiveEventLoopGroup.java:46) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.doExecute(BlockingTaskAwareExecutorServiceImpl.java:169) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.tryBlockedTasks(BlockingTaskAwareExecutorServiceImpl.java:152) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.checkForReadyTasks(BlockingTaskAwareExecutorServiceImpl.java:102) at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl.execute(BlockingTaskAwareExecutorServiceImpl.java:119) at java.util.concurrent.CompletableFuture$UniCompletion.claim(java.base@11.0.10/CompletableFuture.java:568) at java.util.concurrent.CompletableFuture.uniHandle(java.base@11.0.10/CompletableFuture.java:920) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(java.base@11.0.10/CompletableFuture.java:907) at java.util.concurrent.CompletableFuture.postComplete(java.base@11.0.10/CompletableFuture.java:506) at java.util.concurrent.CompletableFuture.complete(java.base@11.0.10/CompletableFuture.java:2073) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.accept(ActionSequencer.java:211) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.accept(ActionSequencer.java:179) at java.util.concurrent.CompletableFuture.uniWhenComplete(java.base@11.0.10/CompletableFuture.java:859) at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(java.base@11.0.10/CompletableFuture.java:883) at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.10/CompletableFuture.java:2251) at java.util.concurrent.CompletableFuture.whenComplete(java.base@11.0.10/CompletableFuture.java:143) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.run(ActionSequencer.java:227) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)