My product running on WildFly 23.01.Final. and I have a script to invoke jboss-cli.sh script to check server status.
In case server is starting and command got timeout with below error, the script hangs:
Failed to connect to the controller: The controller is not available at 30.1.0.152:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out
I even modified AbstractHandleableCloseable.close() to add 30s timeout, but it still hangs, wait(timeout) does not work either.
jstack:
bash-4.2$ <JAVA_HOME_8>/bin/java -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:<WILDFLY_HOME>/bin/jboss-cli-logging.properties -Djboss.remoting.leakdebugging=true -jar <WILDFLY_HOME>/jboss-modules.jar -mp <WILDFLY_HOME>/modules org.jboss.as.cli -c --command-timeout=15 --timeout=50 --controller=30.1.0.152:9990 --command='read-attribute server-state'
Failed to connect to the controller: The controller is not available at 30.1.0.152:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out
2021-08-26 15:29:12
Full thread dump OpenJDK 64-Bit Server VM (25.292-b10 mixed mode):
"CLI Shutdown Hook" #12 prio=5 os_prio=0 tid=0x00007f4db055b800 nid=0x3215 in Object.wait() [0x00007f4d5cc35000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000649e98128> (a java.lang.Object)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:188) - locked <0x0000000649e98128> (a java.lang.Object)
at org.jboss.as.cli.impl.CLIModelControllerClient$1.shutdown(CLIModelControllerClient.java:92)
at org.jboss.as.cli.impl.CliShutdownHook$1.run(CliShutdownHook.java:50) - locked <0x0000000649eb4500> (a java.util.ArrayList)
at java.lang.Thread.run(Thread.java:748)
"Remoting "cli-client" Accept" #14 daemon prio=5 os_prio=0 tid=0x00007f4db1003800 nid=0x320d runnable [0x00007f4d5d2b1000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000064a0d9458> (a sun.nio.ch.Util$3)
- locked <0x000000064a0d9468> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000064a0d9410> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)
"Remoting "cli-client" I/O-1" #13 daemon prio=5 os_prio=0 tid=0x00007f4db0d30800 nid=0x320c runnable [0x00007f4d9015c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000064a0daf80> (a sun.nio.ch.Util$3)
- locked <0x000000064a0db248> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000064a0daf00> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)
"Reference Reaper" #10 daemon prio=5 os_prio=0 tid=0x00007f4db035f800 nid=0x3011 in Object.wait() [0x00007f4d90aed000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000649da94a8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x0000000649da94a8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at org.jboss.modules.ref.References$ReaperThread.run(References.java:64)
"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f4db0272000 nid=0x2fec runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4db0264800 nid=0x2feb waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4db0262000 nid=0x2fea waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4db0260800 nid=0x2fe9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4db025d000 nid=0x2fe8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4db025a000 nid=0x2fe7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4db0227000 nid=0x2fe3 in Object.wait() [0x00007f4d91721000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000064a281140> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) - locked <0x000000064a281140> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4db0222800 nid=0x2fe1 in Object.wait() [0x00007f4d91822000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000064a281180> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191) - locked <0x000000064a281180> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007f4db004c000 nid=0x2f97 in Object.wait() [0x00007f4db7fb3000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x000000064a6b7210> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1252) - locked <0x000000064a6b7210> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:170)
at java.lang.Shutdown.exit(Shutdown.java:216) - locked <0x000000064a587920> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Runtime.exit(Runtime.java:109)
at java.lang.System.exit(System.java:973)
at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:329)
at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
at java.lang.invoke.LambdaForm$DMH/245257410.invokeStatic_L_V(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$MH/1663619914.invokeExact_MT(LambdaForm$MH)
at org.jboss.modules.Module.run(Module.java:353)
at org.jboss.modules.Module.run(Module.java:321)
at org.jboss.modules.Main.main(Main.java:617)
"VM Thread" os_prio=0 tid=0x00007f4db0219000 nid=0x2fe0 runnable
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4db005f000 nid=0x2fd0 runnable
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4db0060800 nid=0x2fd2 runnable
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4db0062800 nid=0x2fd3 runnable
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4db0064800 nid=0x2fd4 runnable
"GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4db0066800 nid=0x2fd6 runnable
"GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4db0068000 nid=0x2fd8 runnable
"GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4db006a000 nid=0x2fd9 runnable
"GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4db006c000 nid=0x2fda runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007f4db0274800 nid=0x2fed waiting on condition
JNI global references: 423
Heap
PSYoungGen total 208384K, used 8865K [0x0000000743400000, 0x0000000750f00000, 0x00000007c0000000)
eden space 192512K, 4% used [0x0000000743400000,0x0000000743ca87a0,0x000000074f000000)
from space 15872K, 0% used [0x000000074ff80000,0x000000074ff80000,0x0000000750f00000)
to space 15872K, 0% used [0x000000074f000000,0x000000074f000000,0x000000074ff80000)
ParOldGen total 143360K, used 11958K [0x0000000649c00000, 0x0000000652800000, 0x0000000743400000)
object space 143360K, 8% used [0x0000000649c00000,0x000000064a7ad980,0x0000000652800000)
Metaspace used 20408K, capacity 21838K, committed 22016K, reserved 1069056K
class space used 2585K, capacity 3019K, committed 3072K, reserved 1048576K