Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Critical
Component/s: Clustering
Labels:
None

We hit an issue when running tests for RFE EAP7-942.

Wildfly was built from https://github.com/rhusar/wildfly/tree/WFLY-9294

Scenario:
2-node Wildfly cluster + 2-node JDG 7.2.0.CR2 cluster.
Wildfly servers are configured to use hotrod store which is backed up by JDG cluster.
Each Wildfly node is gracefully shut down and then started up again.

Wildfly configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf18/standalone-ha.xml
JDG configuration:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/artifact/report/config/jboss-perf20/clustered.xml

Important configuration change on JDG: default cache has transactions configured

<distributed-cache name="default">
    <transaction mode="NON_DURABLE_XA" locking="PESSIMISTIC"/>
</distributed-cache>

With this change, the offloading scenario fails to start. When clusterbench is deployed on EAP and the distributed cache is created on JDG, JDG throws

[JBossINF] [0m[31m04:32:27,869 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (remote-thread--p2-t16) ISPN000136: Error executing command PutKeyValueCommand, writing keys [ClusterConfigurationScope{scope='cache', name='clusterbench-ee7.ear.cb.war'}]: org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
[JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:401)
[JBossINF] 	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
[JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:419)
[JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1565)
[JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1561)
[JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1556)
[JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1535)
[JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryCreated(CacheNotifierImpl.java:368)
[JBossINF] 	at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:261)
[JBossINF] 	at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitSingleEntry(ClusteringDependentLogic.java:493)
[JBossINF] 	at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:186)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:578)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:746)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:555)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.applyChanges(EntryWrappingInterceptor.java:611)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$setSkipRemoteGetsAndInvokeNextForDataCommand$8(EntryWrappingInterceptor.java:667)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:108)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:664)
[JBossINF] 	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:304)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:107)
[JBossINF] 	at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:39)
[JBossINF] 	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:77)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268)
[JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:118)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:57)
[JBossINF] 	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:171)
[JBossINF] 	at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:126)
[JBossINF] 	at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:92)
[JBossINF] 	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
[JBossINF] 	at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
[JBossINF] 	at org.infinispan.interceptors.DDAsyncInterceptor.visitPutKeyValueCommand(DDAsyncInterceptor.java:59)
[JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:79)
[JBossINF] 	at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
[JBossINF] 	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:267)
[JBossINF] 	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommandAsync(BaseRpcInvokingCommand.java:71)
[JBossINF] 	at org.infinispan.commands.remote.SingleRpcCommand.invokeAsync(SingleRpcCommand.java:58)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:95)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:91)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:69)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:41)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JBossINF] 	at java.lang.Thread.run(Thread.java:748)
[JBossINF] Caused by: java.nio.channels.OverlappingFileLockException
[JBossINF] 	at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
[JBossINF] 	at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
[JBossINF] 	at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1063)
[JBossINF] 	at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
[JBossINF] 	at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.storeAll(OverlayLocalConfigurationStorage.java:92)
[JBossINF] 	at org.infinispan.globalstate.impl.OverlayLocalConfigurationStorage.createCache(OverlayLocalConfigurationStorage.java:47)
[JBossINF] 	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.createCacheLocally(GlobalConfigurationManagerImpl.java:159)
[JBossINF] 	at org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(GlobalConfigurationStateListener.java:28)
[JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:498)
[JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:396)
[JBossINF] 	... 50 more

JDG log:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/console-perf21/

This results in EAP logging

[JBossINF] [0m[33m04:32:27,966 WARN  [org.infinispan.client.hotrod.impl.protocol.Codec21] (HotRod-client-async-pool-5) ISPN004005: Error received from the server: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf20, see cause for remote stack trace
[JBossINF] org.infinispan.remoting.RemoteException: ISPN000217: Received exception from perf21, see cause for remote stack trace
[JBossINF] org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [java.nio.channels.OverlappingFileLockException] while invoking method [public void org.infinispan.globalstate.impl.GlobalConfigurationStateListener.createCache(org.infinispan.notifications.cachelistener.event.CacheEntryCreatedEvent)] on listener instance: org.infinispan.globalstate.impl.GlobalConfigurationStateListener@24563360
[JBossINF] java.nio.channels.OverlappingFileLockException
[JBossINF] [0m[31m04:32:27,970 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 74) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.web.clusterbench-ee7.ear.cb.war: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
[JBossINF] 	at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:70)
[JBossINF] 	at org.wildfly.clustering.service.AsynchronousServiceBuilder.lambda$start$0(AsynchronousServiceBuilder.java:99)
[JBossINF] 	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
[JBossINF] 	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
[JBossINF] 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
[JBossINF] 	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1349)
[JBossINF] 	at java.lang.Thread.run(Thread.java:748)
[JBossINF] 	at org.jboss.threads.JBossThread.run(JBossThread.java:485)
[JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl
[JBossINF] 	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83)
[JBossINF] 	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
[JBossINF] 	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
[JBossINF] 	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
[JBossINF] 	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
[JBossINF] 	at org.infinispan.security.Security.doPrivileged(Security.java:76)
[JBossINF] 	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
[JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
[JBossINF] 	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:229)
[JBossINF] 	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1020)
[JBossINF] 	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421)
[JBossINF] 	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644)
[JBossINF] 	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589)
[JBossINF] 	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475)
[JBossINF] 	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461)
[JBossINF] 	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447)
[JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:86)
[JBossINF] 	at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:71)
[JBossINF] 	at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.get(CacheBuilder.java:54)
[JBossINF] 	at org.wildfly.clustering.service.SuppliedValueService.lambda$new$0(SuppliedValueService.java:42)
[JBossINF] 	at org.wildfly.clustering.service.FunctionalValueService.start(FunctionalValueService.java:68)
[JBossINF] 	... 7 more
[JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders
[JBossINF] 	at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179)
[JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:498)
[JBossINF] 	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
[JBossINF] 	... 31 more

EAP log:
https://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-failover-remote-jdg-session-shutdown-invalidation-sync-hotrod/45/console-perf19/

is cloned by

ISPN-9146 [pretesting] JDG throws OverlappingFileLockException when creating distributed cache with transactions in session offloading scenario with hotrod store configured

Closed

Assignee:: Radoslav Husar

Reporter:: Michal Vinkler

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Created:: 2018/05/11 5:13 AM

Updated:: 2021/10/24 6:27 AM

Resolved:: 2018/05/11 3:40 PM

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates