Loading...

Details

Type: Bug
Resolution: Obsolete
Priority: Critical
Fix Version/s: 11.0.0.Alpha2
Affects Version/s: 9.2.0.CR1
Component/s: None
Labels:
None

Git Pull Request:
https://github.com/infinispan/infinispan/pull/5699, https://github.com/infinispan/infinispan/pull/5702

Description

This was observed during the execution of test HotRodRemoteStreamingIT, Line 499 on method stopServer().

The surefire process waits forever for the two servers to stop.

One of the servers is waiting for the initial state transfer to complete:

"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fdaf48f69f0 nid=0x590a waiting on condition [0x00007fdae1a26000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee8cfc00> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
        at org.infinispan.commons.util.SecurityActions$$Lambda$170/1509149284.run(Unknown Source)
        at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
        at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
        at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
        at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1225674479.run(Unknown Source)
        at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
        at org.infinispan.factories.SecurityActions$$Lambda$176/1784470267.run(Unknown Source)
        at org.infinispan.security.Security.doPrivileged(Security.java:76)
        at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
        at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
        at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
        at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
        at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
        at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
        at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/218224883.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:748)

The other server is running a series of operations after the server stops, such
as EmbeddedCacheManager#undefineConfiguration, this can be seen on 3 threads:
MSC service thread 1-1, MSC service thread 1-2 and MSC service thread 1-3

"MSC service thread 1-1" #12 prio=5 os_prio=0 tid=0x00007fd4e86720a0 nid=0x56f1 waiting on condition [0x00007fd4bc8f2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000fb922a18> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729)
        at java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1934)
        at org.infinispan.manager.DefaultCacheManager.undefineConfiguration(DefaultCacheManager.java:394)
        at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:25)
        at org.infinispan.security.actions.UndefineConfigurationAction.run(UndefineConfigurationAction.java:13)
        at org.infinispan.security.Security.doPrivileged(Security.java:76)
        at org.infinispan.registry.impl.SecurityActions.undefineConfiguration(SecurityActions.java:39)
        at org.infinispan.registry.impl.InternalCacheRegistryImpl.unregisterInternalCache(InternalCacheRegistryImpl.java:77)
        - locked <0x00000000ee2e1420> (a org.infinispan.registry.impl.InternalCacheRegistryImpl)
        at org.infinispan.server.hotrod.HotRodServer.stop(HotRodServer.java:486)
        at org.infinispan.server.endpoint.subsystem.ProtocolServerService.doStop(ProtocolServerService.java:216)
        at org.infinispan.server.endpoint.subsystem.ProtocolServerService.stop(ProtocolServerService.java:206)
        - locked <0x00000000edd0eae0> (a org.infinispan.server.endpoint.subsystem.ProtocolServerService)
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:2056)
        at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:2017)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

For some reason, also in the other server, the CounterModule lifecycle is reacting to a
cache manager start event, and it's hanged as well:

"Thread-48" #95 prio=5 os_prio=0 tid=0x00007fd4986a2090 nid=0x575b waiting on condition [0x00007fd48f2f4000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ee91b048> (a java.util.concurrent.CountDownLatch$Sync)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
        at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
        at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:231)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
        at org.infinispan.commons.util.SecurityActions$$Lambda$170/1758042552.run(Unknown Source)
        at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
        at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
        at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
        at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
        at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
        at org.infinispan.factories.AbstractComponentRegistry$$Lambda$175/1042194083.run(Unknown Source)
        at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
        at org.infinispan.factories.SecurityActions$$Lambda$176/2118421355.run(Unknown Source)
        at org.infinispan.security.Security.doPrivileged(Security.java:76)
        at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
        at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
        at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:228)
        at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:962)
        at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411)
        at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:637)
        at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:582)
        at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:468)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454)
        at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:440)
        at org.infinispan.counter.impl.CounterModuleLifecycle.lambda$startCaches$0(CounterModuleLifecycle.java:132)
        at org.infinispan.counter.impl.CounterModuleLifecycle$$Lambda$194/1023465788.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:748)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

s1
95 kB
2018/01/25 9:00 AM
s2
102 kB
2018/01/25 9:00 AM
test
722 kB
2018/01/25 9:00 AM

Server hangs during stop

Details

Description

Attachments

Attachments

Activity

People

Dates