Hi Howard,

I tried following test scenario:

1. Start cluster of 4 nodes - A,B,C,D
2. Start 4 publishers and 4 subscribers on one clustered topic - clients use ClusteredConnectionFactory with client_ack session.
3. Kill server A
4. Kill server B
5. Kill server C
6. Start A
7. Kill D
8. Stop producers

Clients should be reconnected to other nodes in cluster but during killing server sometimes happens that client won't get reconnected to other node and ends with exception. I see following exception in client log:

Consumer for node: 10.16.90.68:1099(it's connected using clustered connection factory -> it does not mean that it's really connected to perf25) and topic /topic/testDistributedTopic ENDED because of exception. Number of consumed messages is 5511 , receiveTimeout=120000, contextProperties=

{java.naming.provider.url=jnp://10.16.90.68:1099, java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}

</b>
Print the exception:
A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[ke-gzdmxmog-1-7m5exmog-ba63px-w1i2ga] recoverDeliveries [oe-00emxmog-1-7m5exmog-ba63px-w1i2ga]
org.jboss.jms.exception.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[ke-gzdmxmog-1-7m5exmog-ba63px-w1i2ga] recoverDeliveries [oe-00emxmog-1-7m5exmog-ba63px-w1i2ga]
at org.jboss.messaging.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72)
at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:675)
at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

Logs from servers attached. Also there is hudson job:
https://hudson.qa.jboss.com/hudson/view/EAP5-JBM-Failover/job/eap-5-topic-client-kill/3/

Could you look at it, please? Give me know if you need more information.

Thank you,

Mirek