Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-3590

Hang in ServerStartFailureTestCase

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 4.0.0.Beta2
    • 4.0.0.Alpha9
    • Management, Server
    • None

      Hang observed in https://ci.wildfly.org/viewLog.html?buildId=88611&buildTypeId=WildFlyCore_PullRequest&tab=buildLog&_focus=5832

      I'll attach the thread dump.

      dlloyd@redhat.com I assigned this to you mostly as a form of ping, as I want to talk to you about it and you are away today.

      Interesting parts of the thread dump:

      "Thread-2" #11 prio=5 os_prio=0 tid=0xe13f0400 nid=0x4c49 waiting on condition [0xde4ed000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0xe5ea9de8> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
	at org.jboss.as.server.BootstrapImpl$ShutdownHook.shutdown(BootstrapImpl.java:276)
	at org.jboss.as.server.BootstrapImpl$ShutdownHook.run(BootstrapImpl.java:240)

"Controller Boot Thread" #25 prio=5 os_prio=0 tid=0xe0ca4c00 nid=0x4c35 waiting for monitor entry [0xdf3fe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at java.lang.Shutdown.exit(Shutdown.java:212)
	- waiting to lock <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Runtime.exit(Runtime.java:109)
	at java.lang.System.exit(System.java:971)
	at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
	at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
	at org.jboss.as.server.ServerService.boot(ServerService.java:405)
	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:370)
	at java.lang.Thread.run(Thread.java:748)

"MSC service thread 1-8" #20 prio=5 os_prio=0 tid=0x087b8c00 nid=0x4c2f in Object.wait() [0xe03ba000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
	at java.lang.Object.wait(Object.java:502)
	at org.jboss.msc.service.Lockable.acquireWrite(Lockable.java:97)
	at org.jboss.msc.service.ServiceControllerImpl$RemoveTask.execute(ServiceControllerImpl.java:1865)
	- locked <0xe32a3f28> (a org.jboss.msc.service.ServiceRegistrationImpl)
	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1527)
	at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
	at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
	at org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788)
	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1537)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1979)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1481)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1374)
	at java.lang.Thread.run(Thread.java:748)

"main" #1 prio=5 os_prio=0 tid=0xf6509000 nid=0x4c02 in Object.wait() [0xf6685000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
	at java.lang.Thread.join(Thread.java:1252)
	- locked <0xe32b58e8> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
	at java.lang.Thread.join(Thread.java:1326)
	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
	at java.lang.Shutdown.runHooks(Shutdown.java:123)
	at java.lang.Shutdown.sequence(Shutdown.java:167)
	at java.lang.Shutdown.exit(Shutdown.java:212)
	- locked <0xe31d5e18> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Runtime.exit(Runtime.java:109)
	at java.lang.System.exit(System.java:971)
	at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:117)
	at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:98)
	at org.jboss.as.server.DomainServerMain.main(DomainServerMain.java:183)
	at java.lang.invoke.LambdaForm$DMH/7468253.invokeStatic_L_V(LambdaForm$DMH)
	at java.lang.invoke.LambdaForm$MH/7742980.invokeExact_MT(LambdaForm$MH)
	at org.jboss.modules.Module.runMainMethod(Module.java:348)
	at org.jboss.modules.Module.run(Module.java:328)
	at org.jboss.modules.Main.main(Main.java:557)

      This is a domain server. The "main" thread has recognized that its ProcessController has closed its stdin, so it is shutting down via System.exit.

      "Thread-2" is running BootstrapImpl.ShutdownHook, waiting on a latch for the MSC ServiceContainer to complete termination. So the SC not completing termination is the basic issue.

      "Controller Boot Thread" is there because this termination occurred during boot. That caused some problem during boot (not surprising) so it is responding to that problem by trying to terminate the process, via System.exit. It's blocking waiting for "main" which has done the same. This thread should not be preventing MSC terminating though; it's not, for example called as part of a StartContext.asynchronous thing. IOW I don't think this thread is relevant to the problem.

      "MSC service thread 1-8" is the most interesting one to me. An MSC thread is blocked but it's not clear to me why. An interesting frame in the stack is org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:788). That shows that ServiceControllerImpl$RemoveTask was passed to the executor but a RejectedExecutionException was thrown, so the task is being run from the thread that attempted to pass it to the executor. Should the MSC executor be rejecting tasks before all service controllers are removed?

        1. WFCORE-3590-threads.txt
          74 kB

              ropalka Richard Opalka
              bstansbe@redhat.com Brian Stansberry
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: