Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-7092

Failed to broadcast asynchronous command

XMLWordPrintable

    • DataGrid Sprint #30

      Occured on server at the end of server stopping.

      Scenarios affected:
      eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync
      eap-7x-failover-ejb-ejbremote-jvmkill-repl-async
      eap-7x-failover-ejb-ejbremote-jvmkill-repl-sync
      eap-7x-failover-ejb-ejbremote-shutdown-dist-async
      eap-7x-failover-ejb-ejbremote-shutdown-dist-sync-tcpStack
      eap-7x-failover-ejb-ejbremote-shutdown-repl-sync
      eap-7x-failover-ejb-ejbremote-undeploy-repl-sync
      eap-7x-failover-ejb-ejbservlet-shutdown-repl-async
      eap-7x-failover-ejb-ejbstateless-jvmkill-repl-async
      eap-7x-failover-http-granular-shutdown-repl-sync
      eap-7x-failover-http-session-shutdown-repl-async
      eap-7x-failover-http-session-shutdown-repl-async-tcpStack
      eap-7x-failover-http-session-shutdown-repl-sync
      eap-7x-failover-jvmkill-ha-ss

      WARN message has two variants.
      Variant 1:

      15:01:15,157 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000080: Disconnecting JGroups channel web
[JBossINF] �[0m�[0m15:01:15,157 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-3) ISPN000082: Stopping the RpcDispatcher for channel web
[JBossINF] �[0m�[0m15:01:15,158 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
[JBossINF] �[0m�[33m15:01:15,158 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p7-t13) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev214, joinInfo=null, topologyId=35, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=STABLE_TOPOLOGY_UPDATE, sender=dev213, joinInfo=null, topologyId=36, rebalanceId=17, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 124+132, dev215: 132+124]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastStableTopologyUpdate(ClusterTopologyManagerImpl.java:615)
[JBossINF] 	at org.infinispan.topology.ClusterCacheStatus.startQueuedRebalance(ClusterCacheStatus.java:633)
[JBossINF] 	at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:373)
[JBossINF] 	at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
[JBossINF] 	at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
[JBossINF] 	at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
[JBossINF] 	at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] Caused by: org.infinispan.commons.CacheException: java.lang.IllegalStateException: channel is not connected
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.rethrowAsCacheException(CommandAwareRpcDispatcher.java:158)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:138)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:578)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
[JBossINF] 	... 13 more
[JBossINF] Caused by: java.lang.IllegalStateException: channel is not connected
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.down(MessageDispatcher.java:682)
[JBossINF] 	at org.jgroups.blocks.RequestCorrelator.sendRequest(RequestCorrelator.java:172)
[JBossINF] 	at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:325)
[JBossINF] 	at org.jgroups.blocks.GroupRequest.sendRequest(GroupRequest.java:76)
[JBossINF] 	at org.jgroups.blocks.Request.execute(Request.java:67)
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:373)
[JBossINF] 	at org.jgroups.blocks.MessageDispatcher.cast(MessageDispatcher.java:386)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processCalls(CommandAwareRpcDispatcher.java:407)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:133)
[JBossINF] 	... 16 more

      Variant 2:

      15:53:27,821 INFO  [org.infinispan.CLUSTER] (remote-thread--p6-t19) ISPN000336: Finished cluster-wide rebalance for cache routing, topology id = 33
[JBossINF] �[0m�[0m15:53:27,829 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
[JBossINF] �[0m�[0m15:53:27,830 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting JGroups channel web
[JBossINF] �[0m�[0m15:53:27,830 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher for channel web
[JBossINF] �[0m�[0m15:53:27,821 INFO  [org.infinispan.CLUSTER] (remote-thread--p6-t14) ISPN000336: Finished cluster-wide rebalance for cache clusterbench-ee7.ear.clusterbench-ee7-web-default.war, topology id = 39
[JBossINF] �[0m�[0m15:53:27,834 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-3) WFLYJCA0019: Stopped Driver service with driver-name = h2
[JBossINF] �[0m�[33m15:53:27,832 WARN  [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread--p6-t14) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=REBALANCE_CONFIRM, sender=dev215, joinInfo=null, topologyId=39, rebalanceId=0, currentCH=null, pendingCH=null, availabilityMode=null, actualMembers=null, throwable=null, viewId=12}: org.infinispan.commons.CacheException: Failed to broadcast asynchronous command: CacheTopologyControlCommand{cache=clusterbench-ee7.ear.clusterbench-ee7-web-default.war, type=CH_UPDATE, sender=dev213, joinInfo=null, topologyId=40, rebalanceId=21, currentCH=DefaultConsistentHash{ns=256, owners = (2)[dev214: 129+127, dev215: 127+129]}, pendingCH=null, availabilityMode=null, actualMembers=[dev214, dev215], throwable=null, viewId=12}
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:595)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.broadcastTopologyUpdate(ClusterTopologyManagerImpl.java:606)
[JBossINF] 	at org.infinispan.topology.ClusterCacheStatus.endRebalance(ClusterCacheStatus.java:369)
[JBossINF] 	at org.infinispan.topology.ClusterCacheStatus.doConfirmRebalance(ClusterCacheStatus.java:310)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:263)
[JBossINF] 	at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:176)
[JBossINF] 	at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:153)
[JBossINF] 	at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$2.run(GlobalInboundInvocationHandler.java:159)
[JBossINF] 	at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:199)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at org.jboss.as.clustering.infinispan.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:48)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] Caused by: java.lang.NullPointerException
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotelyAsync(JGroupsTransport.java:611)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:532)
[JBossINF] 	at org.infinispan.topology.ClusterTopologyManagerImpl.executeOnClusterAsync(ClusterTopologyManagerImpl.java:592)
[JBossINF] 	... 12 more

      Link to server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-jvmkill-dist-sync/29/console-dev213/

              dberinde@redhat.com Dan Berindei (Inactive)
              rhn-engineering-rhusar Radoslav Husar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: