-
Bug
-
Resolution: Done
-
Major
-
6.0.1.Final
-
None
Our current production system has a cluster of 2 nodes using Wildfly 8.0.1. We have used the "ha" profile and "ha" socket-binding-group as a baseline to create the profile for the clusters. The cluster has a REST application deployed. This application uses 2 infinispan cache containers (hibernate and application). In total these views have 5 local caches, 3 invalidation caches and 7 replicated caches. The cache containers use udp as transport. Our hibernate container is the default config as provided by the "ha" profile.
Last thursday and yesterday the one of the nodes started throwing SuspectException during a hibernate flush. The applications had only a few active users.
Node 1 says:
-----------------
2014-04-15 17:40:53,380 ERROR [org.jboss.as.ejb3.invocation] (default task-22) JBAS014134: EJB Invocation failed on component ConversatieResourceRESTService
for method public javax.ws.rs.core.Response ...ConversatieResourceRESTService.setStatus(...event.Con
versatieStatusObject): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
...
...
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.CacheImpl.put(CacheImpl.java:870) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.DecoratedCache.put(DecoratedCache.java:401) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:276) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.update(TransactionalAccessDelegate.java:192) [hibernate-infinispan-4.3.4.Final.j
ar:4.3.4.Final]
at org.hibernate.cache.infinispan.entity.TransactionalAccess.update(TransactionalAccess.java:89) [hibernate-infinispan-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.action.internal.EntityUpdateAction.cacheUpdate(EntityUpdateAction.java:234) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:209) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:461) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:347) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350) [hibernate-core-4.3.4.Final.j
ar:4.3.4.Final]
at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallba
ckCoordinatorNonTrackingImpl.java:110) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
... 117 more
Caused by: SuspectedException
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:406)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353) [infinispan-core-6.0.1.F
inal.jar:6.0.1.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) [infinispan-core-6.0.1
.Final.jar:6.0.1.Final]
... 161 more
2014-04-15 17:40:53,707 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/hibernate-even|2] (1) [node1:rest1-even/hibernate-even]
2014-04-15 17:40:53,707 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-14,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/application-even|2] (1) [node1:rest1-even/application-even]
Node 2 says:
-----------------
2014-04-15 17:40:17,934 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/hibernate-even|2] (1) [node2:rest2-even/hibernate-even]
2014-04-15 17:40:17,934 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/application-even|2] (1) [node2:rest2-even/application-even]
...
...
2014-04-15 17:40:50,737 WARN [org.jgroups.protocols.UDP] (Timer-2,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/hibernate-even, dropping message
2014-04-15 17:40:52,543 WARN [org.jgroups.protocols.UDP] (Timer-5,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/application-even, dropping message
(repeats 100 times)
...
Is there any way I can get more logging than the WARNs above? Does anyone have pointers how or when this SuspectException is thrown?
- is related to
-
ISPN-6599 PutAll operation in the Hot Rod client only partially completed during topology changes
- Closed