Details
-
Bug
-
Resolution: Done
-
Critical
-
None
Description
We hit an issue when running tests for RFE EAP7-942.
Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294
Scenario:
2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
Each Wildfly node is gracefully shut down and then started up again.
Wildfly configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf18/standalone-ha.xml
JDG configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf20/clustered.xml
Important configuration change on JDG: default cache has transactions configured
<distributed-cache name="default"> <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/> </distributed-cache>
With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws
[JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
[JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
[JBossINF] at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
[JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
[JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
[JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
[JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
[JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
[JBossINF] at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
[JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
[JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
[JBossINF] at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
[JBossINF] at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
[JBossINF] at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
[JBossINF] at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
[JBossINF] at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
[JBossINF] at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
[JBossINF] at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
[JBossINF] at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
[JBossINF] at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
[JBossINF] at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
[JBossINF] at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
[JBossINF] at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
[JBossINF] at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
[JBossINF] at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
[JBossINF] at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
[JBossINF] at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
[JBossINF] at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
[JBossINF] at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
[JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
[JBossINF] at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
[JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
[JBossINF] at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
[JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] at java.lang.reflect.Method.invoke(Method.java:498)
[JBossINF] at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
[JBossINF] ... 50 more
This results in EAP logging
[JBossINF] [0m[33m04:32:27,966 WARN [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace [JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace [JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360 [JBossINF] java.nio.channels.OverlappingFileLockException [JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70) [JBossINF] at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99) [JBossINF] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487) [JBossINF] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349) [JBossINF] at java.lang.Thread.run(Thread.java:748) [JBossINF] at org.jboss.threads.JBossThread.run(JBossThread.java:485) [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) [JBossINF] at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) [JBossINF] at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) [JBossINF] at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) [JBossINF] at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) [JBossINF] at org.infinispan.security.Security.doPrivileged(Security.java:76) [JBossINF] at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) [JBossINF] at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) [JBossINF] at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229) [JBossINF] at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020) [JBossINF] at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) [JBossINF] at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) [JBossINF] at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) [JBossINF] at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) [JBossINF] at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) [JBossINF] at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86) [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71) [JBossINF] at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54) [JBossINF] at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42) [JBossINF] at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68) [JBossINF] ... 7 more [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders [JBossINF] at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179) [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [JBossINF] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [JBossINF] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [JBossINF] at java.lang.reflect.Method.invoke(Method.java:498) [JBossINF] at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) [JBossINF] ... 31 more
Attachments
Issue Links
- is cloned by
-
-
- Closed
-