Hi,

I tried following test scenario:

1. Start cluster of 4 nodes - A,B,C,D
2. Start 4 Producers and 4 Consumers on one clustered queue - clients use ClusteredConnectionFactory with client_ack session.
3. Kill server A
4. Kill server B
5. Kill server C
6. Start A
7. Kill D
8. Stop producers

Clients should be reconnected to other nodes in cluster but during killing server sometimes happens that client won't get reconnected to other node and ends with exception. I see following exception in server log:

[JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[we-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [8f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
[JBoss] java.util.ConcurrentModificationException
[JBoss] at java.util.LinkedHashMap$LinkedHashIterator.remove(LinkedHashMap.java:364)
[JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:355)
[JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
[JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
[JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
[JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
[JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
[JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[ye-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [6f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
[JBoss] java.util.ConcurrentModificationException
[JBoss] at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
[JBoss] at java.util.LinkedHashMap$ValueIterator.next(LinkedHashMap.java:388)
[JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:347)
[JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
[JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
[JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
[JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
[JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)
[JBoss] 12:17:44,257 ERROR [ExceptionUtil] SessionEndpoint[0f-7mhcheog-1-y9b6heog-sfream-y1i2ga] recoverDeliveries [9f-pmhcheog-1-y9b6heog-sfream-y1i2ga]
[JBoss] java.util.ConcurrentModificationException
[JBoss] at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:373)
[JBoss] at java.util.LinkedHashMap$ValueIterator.next(LinkedHashMap.java:388)
[JBoss] at org.jboss.messaging.core.impl.MessagingQueue.removeStrandedReferences(MessagingQueue.java:347)
[JBoss] at org.jboss.jms.server.endpoint.ServerSessionEndpoint.recoverDeliveries(ServerSessionEndpoint.java:667)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$recoverDeliveries$aop(SessionAdvised.java:150)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised$recoverDeliveries_4196799148363996265.invokeTarget(SessionAdvised$recoverDeliveries_4196799148363996265.java)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
[JBoss] at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
[JBoss] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
[JBoss] at org.jboss.jms.server.endpoint.advised.SessionAdvised.recoverDeliveries(SessionAdvised.java)
[JBoss] at org.jboss.jms.wireformat.SessionRecoverDeliveriesRequest.serverInvoke(SessionRecoverDeliveriesRequest.java:96)
[JBoss] at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:165)
[JBoss] at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:791)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:744)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:586)
[JBoss] at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:234)

It seems that there is access to "recoveryMap = Collections.synchronizedMap(new LinkedHashMap());" during iteration.

Could you look at it please?

Thank you,

Mirek