-
Bug
-
Resolution: Done
-
Major
-
10.0.0.Alpha4
Simple shutdown failover tests are failing due to apparent error in / misconfiguration of the distributed cache.
If one follows the aforementioned steps to reproduce, the following symptoms appear:
- Variations on
SEVERE [org.jgroups.protocols.UDP] (TransferQueueBundler,ee,rhel7gax86-64) JGRP000029: rhel7gax86-64: failed sending message to rhel7gax86-64 (59 bytes): java.io.IOException: Invalid argument, headers: UNICAST3: ACK, seqno=9410, ts=217, UDP: [cluster_name=ee]
- Failures of this kind:
ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p2-t12) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 11 at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92) at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:285) at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:254) at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:130) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97) at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102) at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71) at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49) at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58) at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617) at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579) at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572) at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442) at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297) at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286) at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1180) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1139) at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1105) at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:560) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:201) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:45) at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113) at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:285) at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:166) at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:144) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
- Warnings such as:
WARN [org.infinispan.transaction.impl.TransactionTable] (TxCleanupService,dist,rhel7gax86-64) ISPN000326: Remote transaction GlobalTransaction:<rhel7gax86-64>:9373:remote timed out. Rolling back after 70810 ms
Logs from one of such play & test scenarios are attached: logs.zip, configs.zip.
Any ideas which configuration directive or application setting might be at the bottom of this?
Needless to say any such test passes without any error with EAP 6.4 and 6.3 both with shutdown and undeploy failover scenarios.
Thx for comments.
- clones
-
JBEAP-350 Distributable app causes TransferQueueBundler Invalid argument and Topology errors
- Closed