Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-4213

clustered Wildfly unexpected nodes SuspectException

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 8.2.2.Final
    • 6.0.1.Final
    • Core
    • None

      Our current production system has a cluster of 2 nodes using Wildfly 8.0.1. We have used the "ha" profile and "ha" socket-binding-group as a baseline to create the profile for the clusters. The cluster has a REST application deployed. This application uses 2 infinispan cache containers (hibernate and application). In total these views have 5 local caches, 3 invalidation caches and 7 replicated caches. The cache containers use udp as transport. Our hibernate container is the default config as provided by the "ha" profile.

      Last thursday and yesterday the one of the nodes started throwing SuspectException during a hibernate flush. The applications had only a few active users.

      Node 1 says:
      -----------------
      2014-04-15 17:40:53,380 ERROR [org.jboss.as.ejb3.invocation] (default task-22) JBAS014134: EJB Invocation failed on component ConversatieResourceRESTService
      for method public javax.ws.rs.core.Response ...ConversatieResourceRESTService.setStatus(...event.Con
      versatieStatusObject): javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
      ...
      ...
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:32) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:70) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.CacheImpl.putInternal(CacheImpl.java:878) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.CacheImpl.put(CacheImpl.java:870) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.DecoratedCache.put(DecoratedCache.java:401) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:276) [infinispan-core-6.0.1.Final.jar:6.0.1.Final]
      at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.update(TransactionalAccessDelegate.java:192) [hibernate-infinispan-4.3.4.Final.j
      ar:4.3.4.Final]
      at org.hibernate.cache.infinispan.entity.TransactionalAccess.update(TransactionalAccess.java:89) [hibernate-infinispan-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.action.internal.EntityUpdateAction.cacheUpdate(EntityUpdateAction.java:234) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:209) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:461) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:347) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:350) [hibernate-core-4.3.4.Final.j
      ar:4.3.4.Final]
      at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:56) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1222) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:425) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      at org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorNonTrackingImpl.beforeCompletion(SynchronizationCallba
      ckCoordinatorNonTrackingImpl.java:110) [hibernate-core-4.3.4.Final.jar:4.3.4.Final]
      ... 117 more
      Caused by: SuspectedException
      at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:406)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353) [infinispan-core-6.0.1.F
      inal.jar:6.0.1.Final]
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167) [infinispan-core-6.0.1
      .Final.jar:6.0.1.Final]
      ... 161 more
      2014-04-15 17:40:53,707 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-7,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/hibernate-even|2] (1) [node1:rest1-even/hibernate-even]
      2014-04-15 17:40:53,707 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-14,shared=udp) ISPN000094: Received new cluster view: [node1:rest1-even/application-even|2] (1) [node1:rest1-even/application-even]

      Node 2 says:
      -----------------
      2014-04-15 17:40:17,934 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-19,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/hibernate-even|2] (1) [node2:rest2-even/hibernate-even]
      2014-04-15 17:40:17,934 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,shared=udp) ISPN000094: Received new cluster view: [node2:rest2-even/application-even|2] (1) [node2:rest2-even/application-even]
      ...
      ...
      2014-04-15 17:40:50,737 WARN [org.jgroups.protocols.UDP] (Timer-2,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/hibernate-even, dropping message
      2014-04-15 17:40:52,543 WARN [org.jgroups.protocols.UDP] (Timer-5,shared=udp) JGRP000032: null: no physical address for node1:rest1-even/application-even, dropping message
      (repeats 100 times)
      ...

      Is there any way I can get more logging than the WARNs above? Does anyone have pointers how or when this SuspectException is thrown?

              Unassigned Unassigned
              hielkehoeve Hielke Hoeve (Inactive)
              Votes:
              9 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: