Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7439

InitialClusterSizeTest can hang during teardown

    Details

      Description

      Test testInitialClusterSizeFail expects the nodes to time out in JGroupsTransport.waitForInitialNodes(), but in at least one case the timeout didn't happen. The test then tried to shut down the cache managers, but it hanged because another thread was holding the GlobalComponentRegistry lock:

      "testng-InitialClusterSizeTest" #13 prio=5 os_prio=0 tid=0x00007f1874d1f000 nid=0x1778 waiting for monitor entry [0x00007f181bafc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      	at org.infinispan.factories.GlobalComponentRegistry.stop(GlobalComponentRegistry.java:280)
      	- waiting to lock <0x0000000093c7afe0> (a org.infinispan.factories.GlobalComponentRegistry)
      	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:701)
      	- locked <0x000000008a005b80> (a org.infinispan.manager.DefaultCacheManager)
      	at org.infinispan.test.TestingUtil.killCacheManagers(TestingUtil.java:656)
      	at org.infinispan.test.MultipleCacheManagersTest.clearContent(MultipleCacheManagersTest.java:138)
      	at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:84)
      	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:564)
      	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
      	at org.testng.internal.Invoker.invokeMethod(Invoker.java:786)
      	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:901)
      	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1231)
      	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
      	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
      	at org.testng.TestRunner.privateRun(TestRunner.java:767)
      	at org.testng.TestRunner.run(TestRunner.java:617)
      	at org.testng.SuiteRunner.runTest(SuiteRunner.java:348)
      	at org.testng.SuiteRunner.access$000(SuiteRunner.java:38)
      	at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:382)
      	at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      "ForkThread-4,InitialClusterSizeTest" #167842 prio=5 os_prio=0 tid=0x00007f1824163800 nid=0x3316 waiting on condition [0x00007f17e62b9000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.jgroups.util.Util.sleep(Util.java:1818)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.firstOfAllClients(ClientGmsImpl.java:181)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:97)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:41)
      	at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1066)
      	at org.jgroups.protocols.tom.TOA.down(TOA.java:73)
      	at org.jgroups.protocols.FlowControl.down(FlowControl.java:302)
      	at org.jgroups.protocols.RSVP.down(RSVP.java:102)
      	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:900)
      	at org.jgroups.JChannel.down(JChannel.java:644)
      	at org.jgroups.JChannel._connect(JChannel.java:873)
      	at org.jgroups.JChannel.connect(JChannel.java:369)
      	- locked <0x0000000093c7aea0> (a org.jgroups.JChannel)
      	at org.jgroups.JChannel.connect(JChannel.java:360)
      	- locked <0x0000000093c7aea0> (a org.jgroups.JChannel)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:221)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:211)
      	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547)
      	- locked <0x0000000093c7afe0> (a org.infinispan.factories.GlobalComponentRegistry)
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:244)
      	- locked <0x0000000093c7afe0> (a org.infinispan.factories.GlobalComponentRegistry)
      	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:666)
      	at org.infinispan.remoting.transport.InitialClusterSizeTest.lambda$testInitialClusterSize$799(InitialClusterSizeTest.java:47)
      	at org.infinispan.remoting.transport.InitialClusterSizeTest$$Lambda$2092/593962598.run(Unknown Source)
      	at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:510)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      "ForkThread-2,InitialClusterSizeTest" #167840 prio=5 os_prio=0 tid=0x00007f1824164800 nid=0x3314 waiting on condition [0x00007f17eec40000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
      	at java.lang.Thread.sleep(Native Method)
      	at org.jgroups.util.Util.sleep(Util.java:1818)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.firstOfAllClients(ClientGmsImpl.java:181)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:97)
      	at org.jgroups.protocols.pbcast.ClientGmsImpl.join(ClientGmsImpl.java:41)
      	at org.jgroups.protocols.pbcast.GMS.down(GMS.java:1066)
      	at org.jgroups.protocols.tom.TOA.down(TOA.java:73)
      	at org.jgroups.protocols.FlowControl.down(FlowControl.java:302)
      	at org.jgroups.protocols.RSVP.down(RSVP.java:102)
      	at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:900)
      	at org.jgroups.JChannel.down(JChannel.java:644)
      	at org.jgroups.JChannel._connect(JChannel.java:873)
      	at org.jgroups.JChannel.connect(JChannel.java:369)
      	- locked <0x0000000093c7b7f0> (a org.jgroups.JChannel)
      	at org.jgroups.JChannel.connect(JChannel.java:360)
      	- locked <0x0000000093c7b7f0> (a org.jgroups.JChannel)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:221)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:211)
      	at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:867)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:633)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:622)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:547)
      	- locked <0x0000000093c7b930> (a org.infinispan.factories.GlobalComponentRegistry)
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:244)
      	- locked <0x0000000093c7b930> (a org.infinispan.factories.GlobalComponentRegistry)
      	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:666)
      	at org.infinispan.remoting.transport.InitialClusterSizeTest.lambda$testInitialClusterSize$799(InitialClusterSizeTest.java:47)
      	at org.infinispan.remoting.transport.InitialClusterSizeTest$$Lambda$2092/593962598.run(Unknown Source)
      	at org.infinispan.test.AbstractInfinispanTest$RunnableWrapper.run(AbstractInfinispanTest.java:510)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      See the full thread dump here: http://ci.infinispan.org/viewLog.html?buildId=49393&buildTypeId=bt9

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                dan.berindei Dan Berindei
                Reporter:
                dan.berindei Dan Berindei
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: