Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-15170

jboss-cli.sh script hang forever if connection timeout

    XMLWordPrintable

Details

    • Bug
    • Status: Open (View Workflow)
    • Minor
    • Resolution: Unresolved
    • 23.0.1.Final
    • None
    • CLI, Remoting
    • None
    • Hide

      Simply specific --timeout value small enough to check status running server:

      jboss-cli.sh --connect --command-timeout=15 --timeout=5 --command='read-attribute server-state'

      It will hang.

      Show
      Simply specific --timeout value small enough to check status running server: jboss-cli.sh --connect --command-timeout=15 --timeout=5 --command='read-attribute server-state' It will hang.

    Description

      My product running on WildFly 23.01.Final. and I have a script to invoke jboss-cli.sh script to check server status.

      In case server is starting and command got timeout with below error, the script hangs:

      Failed to connect to the controller: The controller is not available at 30.1.0.152:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out

      I even modified AbstractHandleableCloseable.close() to add 30s timeout, but it still hangs, wait(timeout) does not work either.

      jstack:

      bash-4.2$ <JAVA_HOME_8>/bin/java -Djboss.modules.system.pkgs=com.sun.java.swing -Dcom.ibm.jsse2.overrideDefaultTLS=true -Dlogging.configuration=file:<WILDFLY_HOME>/bin/jboss-cli-logging.properties -Djboss.remoting.leakdebugging=true -jar <WILDFLY_HOME>/jboss-modules.jar -mp <WILDFLY_HOME>/modules org.jboss.as.cli -c --command-timeout=15 --timeout=50 --controller=30.1.0.152:9990 --command='read-attribute server-state'
      Failed to connect to the controller: The controller is not available at 30.1.0.152:9990: java.net.ConnectException: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out: WFLYPRT0023: Could not connect to remote+http://30.1.0.152:9990. The connection timed out
      2021-08-26 15:29:12
      Full thread dump OpenJDK 64-Bit Server VM (25.292-b10 mixed mode):

      "CLI Shutdown Hook" #12 prio=5 os_prio=0 tid=0x00007f4db055b800 nid=0x3215 in Object.wait() [0x00007f4d5cc35000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x0000000649e98128> (a java.lang.Object)
        at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:188)
      • locked <0x0000000649e98128> (a java.lang.Object)
        at org.jboss.as.cli.impl.CLIModelControllerClient$1.shutdown(CLIModelControllerClient.java:92)
        at org.jboss.as.cli.impl.CliShutdownHook$1.run(CliShutdownHook.java:50)
      • locked <0x0000000649eb4500> (a java.util.ArrayList)
        at java.lang.Thread.run(Thread.java:748)

      "Remoting "cli-client" Accept" #14 daemon prio=5 os_prio=0 tid=0x00007f4db1003800 nid=0x320d runnable [0x00007f4d5d2b1000]
      java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

      • locked <0x000000064a0d9458> (a sun.nio.ch.Util$3)
      • locked <0x000000064a0d9468> (a java.util.Collections$UnmodifiableSet)
      • locked <0x000000064a0d9410> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)

      "Remoting "cli-client" I/O-1" #13 daemon prio=5 os_prio=0 tid=0x00007f4db0d30800 nid=0x320c runnable [0x00007f4d9015c000]
      java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

      • locked <0x000000064a0daf80> (a sun.nio.ch.Util$3)
      • locked <0x000000064a0db248> (a java.util.Collections$UnmodifiableSet)
      • locked <0x000000064a0daf00> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)

      "Reference Reaper" #10 daemon prio=5 os_prio=0 tid=0x00007f4db035f800 nid=0x3011 in Object.wait() [0x00007f4d90aed000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x0000000649da94a8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
      • locked <0x0000000649da94a8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at org.jboss.modules.ref.References$ReaperThread.run(References.java:64)

      "Service Thread" #9 daemon prio=9 os_prio=0 tid=0x00007f4db0272000 nid=0x2fec runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f4db0264800 nid=0x2feb waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f4db0262000 nid=0x2fea waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f4db0260800 nid=0x2fe9 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f4db025d000 nid=0x2fe8 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f4db025a000 nid=0x2fe7 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE

      "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4db0227000 nid=0x2fe3 in Object.wait() [0x00007f4d91721000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x000000064a281140> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
      • locked <0x000000064a281140> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

      "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4db0222800 nid=0x2fe1 in Object.wait() [0x00007f4d91822000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x000000064a281180> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
      • locked <0x000000064a281180> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

      "main" #1 prio=5 os_prio=0 tid=0x00007f4db004c000 nid=0x2f97 in Object.wait() [0x00007f4db7fb3000]
      java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)

      • waiting on <0x000000064a6b7210> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1252)
      • locked <0x000000064a6b7210> (a java.lang.Thread)
        at java.lang.Thread.join(Thread.java:1326)
        at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
        at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
        at java.lang.Shutdown.runHooks(Shutdown.java:123)
        at java.lang.Shutdown.sequence(Shutdown.java:170)
        at java.lang.Shutdown.exit(Shutdown.java:216)
      • locked <0x000000064a587920> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Runtime.exit(Runtime.java:109)
        at java.lang.System.exit(System.java:973)
        at org.jboss.as.cli.impl.CliLauncher.main(CliLauncher.java:329)
        at org.jboss.as.cli.CommandLineMain.main(CommandLineMain.java:45)
        at java.lang.invoke.LambdaForm$DMH/245257410.invokeStatic_L_V(LambdaForm$DMH)
        at java.lang.invoke.LambdaForm$MH/1663619914.invokeExact_MT(LambdaForm$MH)
        at org.jboss.modules.Module.run(Module.java:353)
        at org.jboss.modules.Module.run(Module.java:321)
        at org.jboss.modules.Main.main(Main.java:617)

      "VM Thread" os_prio=0 tid=0x00007f4db0219000 nid=0x2fe0 runnable

      "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4db005f000 nid=0x2fd0 runnable

      "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4db0060800 nid=0x2fd2 runnable

      "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4db0062800 nid=0x2fd3 runnable

      "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4db0064800 nid=0x2fd4 runnable

      "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f4db0066800 nid=0x2fd6 runnable

      "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f4db0068000 nid=0x2fd8 runnable

      "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f4db006a000 nid=0x2fd9 runnable

      "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f4db006c000 nid=0x2fda runnable

      "VM Periodic Task Thread" os_prio=0 tid=0x00007f4db0274800 nid=0x2fed waiting on condition

      JNI global references: 423

      Heap
      PSYoungGen total 208384K, used 8865K [0x0000000743400000, 0x0000000750f00000, 0x00000007c0000000)
      eden space 192512K, 4% used [0x0000000743400000,0x0000000743ca87a0,0x000000074f000000)
      from space 15872K, 0% used [0x000000074ff80000,0x000000074ff80000,0x0000000750f00000)
      to space 15872K, 0% used [0x000000074f000000,0x000000074f000000,0x000000074ff80000)
      ParOldGen total 143360K, used 11958K [0x0000000649c00000, 0x0000000652800000, 0x0000000743400000)
      object space 143360K, 8% used [0x0000000649c00000,0x000000064a7ad980,0x0000000652800000)
      Metaspace used 20408K, capacity 21838K, committed 22016K, reserved 1069056K
      class space used 2585K, capacity 3019K, committed 3072K, reserved 1048576K

       

      Attachments

        1. jboss-cli-success3.log
          44 kB
        2. jboss-cli-logging.properties
          2 kB
        3. jboss-cli-failure3.log
          40 kB
        4. jboss-cli.log
          3 kB
        5. jboss-cli_success.log
          47 kB
        6. jboss-cli_correct.log
          16 kB

        Activity

          People

            flaviarnn Flavia Rainone
            naraganinagaraju Naragani NAGARAJU (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: