IllegalStateException during failover

Hi Howard,

I tried following test scenario:

1. Start cluster of 4 nodes - A,B,C,D
2. Start 4 publishers and 4 subscribers on one clustered topic - clients use ClusteredConnectionFactory with transaction acknowledge session.
3. Kill server A
4. Kill server B
5. Kill server C
6. Start A
7. Kill D
8. Stop producers

Clients should be reconnected to other nodes in cluster during killing servers but sometimes happens that client won't get reconnected to other node and ends with exception. I see following exception in client log:

Consumer for node: 10.16.90.68:1099 and topic /topic/testDistributedTopic ENDED because of exception. Number of consumed messages is 12000 , receiveTimeout=120000, contextProperties=

{java.naming.provider.url=jnp://10.16.90.68:1099, java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces}

</b>
Print the exception:
Cannot find transaction with xid:LocalTx[ft-cpgt3nog-1-bqjh3nog-0fippg-q1i2ga]
javax.jms.IllegalStateException: Cannot find transaction with xid:LocalTx[ft-cpgt3nog-1-bqjh3nog-0fippg-q1i2ga]
at org.jboss.jms.tx.ResourceManager.rollbackLocal(ResourceManager.java:239)
at org.jboss.jms.tx.ResourceManager.commitLocal(ResourceManager.java:223)
at org.jboss.jms.client.container.SessionAspect.handleCommit(SessionAspect.java:582)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleCommit_899770473.invoke(SessionAspect_z_handleCommit_899770473.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.commit(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.commit(JBossSession.java:164)
at org.jboss.smartfrog.jbm.SimpleReceiverTopicTransacted.clientLogic(SimpleReceiverTopicTransacted.java:165)
at org.jboss.smartfrog.jbm.AJMSClient.run(AJMSClient.java:287)
at java.lang.Thread.run(Thread.java:662)
2011/06/07 13:09:34:198 EDT [INFO ][Thread-12] HOST perf21.mw.lab.eng.bos.redhat.com:rootProcess:jbmFailoverCase3 - Component has ended with following result : Termination Record: , type: abnormal, description: Cannot find transaction with xid:LocalTx[ft-cpgt3nog-1-bqjh3nog-0fippg-q1i2ga]
javax.jms.IllegalStateException: Cannot find transaction with xid:LocalTx[ft-cpgt3nog-1-bqjh3nog-0fippg-q1i2ga]
at org.jboss.jms.tx.ResourceManager.rollbackLocal(ResourceManager.java:239)
at org.jboss.jms.tx.ResourceManager.commitLocal(ResourceManager.java:223)
at org.jboss.jms.client.container.SessionAspect.handleCommit(SessionAspect.java:582)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleCommit_899770473.invoke(SessionAspect_z_handleCommit_899770473.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.commit(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.commit(JBossSession.java:164)
at org.jboss.smartfrog.jbm.SimpleReceiverTopicTransacted.clientLogic(SimpleReceiverTopicTransacted.java:165)
at org.jboss.smartfrog.jbm.AJMSClient.run(AJMSClient.java:287)
at java.lang.Thread.run(Thread.java:662)

Please see hudson job:
https://hudson.qa.jboss.com/hudson/view/EAP5-JBM-Failover/job/eap-5-topic-trans-kill/4/

Could you look at it, please? Give me know if you need more information.

Thank you,

Mirek