MessagingNetworkFailureException during failover

Hi Howard,

I tried following test scenario:

1. Start cluster of 4 nodes - A,B,C,D
2. Start 4 publishers and 4 subscribers on one clustered topic - clients use ClusteredConnectionFactory with transaction acknowledge session.
3. Kill server A
4. Kill server B
5. Kill server C
6. Start A
7. Kill D
8. Stop producers

But during killing server sometimes i get following exception:

02:37:44,680 INFO [FailoverCommandCenter] JBoss Messaging server failure detected - waiting for failover to complete...
[JBoss] 02:37:44,688 ERROR [ConnectionTable] failed sending data to 10.16.90.66:7900: java.net.SocketException: Socket closed
[JBoss] 02:37:45,465 ERROR [ClientConsumer] Failed to send changeRate message
[JBoss] org.jboss.jms.exception.MessagingNetworkFailureException
[JBoss] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:240)
[JBoss] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
[JBoss] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
[JBoss] at org.jboss.jms.client.delegate.ClientConsumerDelegate.org$jboss$jms$client$delegate$ClientConsumerDelegate$changeRate$aop(ClientConsumerDelegate.java:138)
[JBoss] at org.jboss.jms.client.delegate.ClientConsumerDelegate$changeRate_N952316153687074823.invokeTarget(ClientConsumerDelegate$changeRate_N952316153687074823.java)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
[JBoss] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
[JBoss] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
[JBoss] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
[JBoss] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
[JBoss] at org.jboss.jms.client.delegate.ClientConsumerDelegate.changeRate(ClientConsumerDelegate.java)
[JBoss] at org.jboss.jms.client.container.ClientConsumer.sendChangeRateMessage(ClientConsumer.java:825)
[JBoss] at org.jboss.jms.client.container.ClientConsumer.resume(ClientConsumer.java:761)
[JBoss] at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.setConsuming(MessageSucker.java:293)
[JBoss] at org.jboss.messaging.core.impl.MessagingQueue.informSuckers(MessagingQueue.java:590)
[JBoss] at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:540)
[JBoss] at org.jboss.messaging.core.impl.ChannelSupport.deliver(ChannelSupport.java:386)
[JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run(ServerSessionEndpoint.java:1693)
[JBoss] at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:104)
[JBoss] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
[JBoss] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
[JBoss] at java.lang.Thread.run(Thread.java:662)
[JBoss] Caused by: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [bisocket://10.16.90.66:4457/?JBM_clientMaxPoolSize=200&clientLeasePeriod=10000&clientSocketClass=org.jboss.jms.client.remoting.ClientSocketWrapper&dataType=jms&failureDisconnectTimeout=0&generalizeSocketException=true&marshaller=org.jboss.jms.wireformat.JMSWireFormat&numberOfCallRetries=5&pingFrequency=30000&pingWindowFactor=71582&socket.check_connection=false&stopLeaseOnFailure=true&timeout=300000&unmarshaller=org.jboss.jms.wireformat.JMSWireFormat&useClientConnectionIdentity=true&validatorPingPeriod=10000&validatorPingTimeout=5000]
[JBoss] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:831)
[JBoss] at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:470)
[JBoss] at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169)
[JBoss] at org.jboss.remoting.Client.invoke(Client.java:2070)
[JBoss] at org.jboss.remoting.Client.invoke(Client.java:879)
[JBoss] at org.jboss.remoting.Client.invoke(Client.java:867)
[JBoss] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
[JBoss] ... 22 more
[JBoss] Caused by: java.net.ConnectException: Connection refused
[JBoss] at java.net.PlainSocketImpl.socketConnect(Native Method)
[JBoss] at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
[JBoss] at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
[JBoss] at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
[JBoss] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
[JBoss] at java.net.Socket.connect(Socket.java:529)
[JBoss] at org.jboss.remoting.transport.socket.SocketClientInvoker.connect(SocketClientInvoker.java:293)
[JBoss] at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:210)
[JBoss] at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:477)
[JBoss] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1165)
[JBoss] at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816)
[JBoss] ... 28 more

This exception was thrown on server on perf25 when EAP server on perf24 was killed.

Hudson job:
https://hudson.qa.jboss.com/hudson/view/EAP5-JBM-Failover/job/eap-5-topic-trans-kill/19

perf21 - deploy host
perf22 - all producers
perf23 - node A + client for node A (was started here but it does not mean it's connected to this node)
perf24 - node B + client for node B (was started here but it does not mean it's connected to this node)
perf25 - node C + client for node C (was started here but it does not mean it's connected to this node)
perf26 - node D + client for node D (was started here but it does not mean it's connected to this node)

Logs are in QA Lab: /home/hudson/users-tmp/mnovak/failover_jmsexception.zip

Could you look at it, please?

Thank you,

Mirek