Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-3047

[EAP][7.2] sporadic exceptions during clustered startup ISPN000329: Unable to read rebalancing status from coordinator <pod>: org.infinispan.remoting.transport.jgroups.SuspectException: ISPN000400: Node <pod> was suspected

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Obsolete
    • Icon: Minor Minor
    • None
    • EAP72 1.0.0.GA
    • EAP7
    • None

      At initial clustered deployment (scale from 0 to 3), the following exception sporadically occurs at startup in the logs.

      It was reproduced with dns.DNS_PING and openshift.KUBE_PING (so probably not ping-specific). The cluster seems to form correctly despite the exception in the logs.

      WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-2) ISPN000329: Unable to read rebalancing status from coordinator hsc-1-hdfml: org.infinispan.remoting.t
      ransport.jgroups.SuspectException: ISPN000400: Node hsc-1-hdfml was suspected
              at org.infinispan.remoting.transport.impl.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:54)
              at org.infinispan.remoting.transport.impl.SingletonMapResponseCollector.targetNotFound(SingletonMapResponseCollector.java:26)
              at org.infinispan.remoting.transport.ValidSingleResponseCollector.addResponse(ValidSingleResponseCollector.java:23)
              at org.infinispan.remoting.transport.impl.SingleTargetRequest.receiveResponse(SingleTargetRequest.java:52)
              at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:35)
              at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:52)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1370)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1273)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:125)
              at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1418)
              at org.jgroups.JChannel.up(JChannel.java:816)
              at org.jgroups.fork.ForkProtocolStack.up(ForkProtocolStack.java:133)
              at org.jgroups.stack.Protocol.up(Protocol.java:340)
              at org.jgroups.protocols.FORK.up(FORK.java:141)
              at org.jgroups.protocols.FRAG3.up(FRAG3.java:171)
              at org.jgroups.protocols.FlowControl.up(FlowControl.java:351)
              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:873)
              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:240)
              at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1003)
              at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:729)
              at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:384)
              at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:130)
              at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:203)
              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:253)
              at org.jgroups.protocols.MERGE3.up(MERGE3.java:280)
              at org.jgroups.protocols.Discovery.up(Discovery.java:269)
              at org.jgroups.protocols.TP.passMessageUp(TP.java:1248)
              at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:87)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at org.jboss.as.clustering.jgroups.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:52)
              at java.lang.Thread.run(Thread.java:748)
              Suppressed: org.infinispan.util.logging.TraceException
                      at org.infinispan.remoting.transport.Transport.invokeRemotely(Transport.java:75)
                      at org.infinispan.topology.ClusterTopologyManagerImpl.fetchRebalancingStatusFromCoordinator(ClusterTopologyManagerImpl.java:144)
                      at org.infinispan.topology.ClusterTopologyManagerImpl.start(ClusterTopologyManagerImpl.java:129)
                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                      at java.lang.reflect.Method.invoke(Method.java:498)
                      at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79)
                      at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71)
                      at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76)
                      at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)
                      at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968)
                      at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703)
                      at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72)
                      at org.infinispan.security.Security.doPrivileged(Security.java:44)
                      at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71)
                      at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696)
                      at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689)
                      at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607)
                      at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:250)
                      at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:680)
                      at org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.get(CacheContainerServiceConfigurator.java:104)
                      at org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.get(CacheContainerServiceConfigurator.java:71)
                      at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67)
                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1738)
                      at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1700)
                      at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1558)
                      at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
                      at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
                      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
                      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1364)
                      ... 1 more
      

            rhn-engineering-rhusar Radoslav Husar
            maschmid@redhat.com Marek Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: