Receiving "Trying to change Tx in enlist!" errors in multi-threaded modules.

Receiving "Trying to change Tx in enlist!" errors in multi-threaded modules. There are three different scenarios in which this error is generated. This error occurs in random and there is no definited pattern. A similar issue was reported and fixed as per the link: http://www.mail-archive.com/jboss-development@lists.sourceforge.net/msg30961.html in an older version of JBoss AS. Any help as early as possbile is hhighly appreciated.

1. Message Driven Bean:

On posting messages to Queue which being associated with a Message Driven bean, the following error is being generated.

21:42:06,084 WARN [org.jboss.resource.connectionmanager.TxConnectionManager] in Enlisting tx, trying to change tx. illegal s
tate: old: TransactionImpl:XidImpl[FormatId=257, GlobalId=DBPHQRPLQ01/112154, BranchQual=, localId=112154], new: TransactionI
mpl:XidImpl[FormatId=257, GlobalId=DBPHQRPLQ01/112155, BranchQual=, localId=112155], cel: org.jboss.resource.connectionmanage
r.TxConnectionManager$TxConnectionEventListener@175b175b[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnecti
on@16c916c9 handles=0 lastUse=1257133326083 permit=true trackByTx=true mcp=org.jboss.resource.connectionmanager.JBossManagedC
onnectionPool$OnePool@3cbc3cbc context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@33373337]
21:42:06,085 ERROR [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Exception in JMSCI message listener
org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.l
ang.IllegalStateException: Trying to change Tx in enlist!)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java
:101)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java
:588)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508)
at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421)
at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:182)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:87)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:94)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:389)
at org.jboss.ejb.Container.invoke(Container.java:873)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:1090)
at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1392)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:256)
at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:904)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:160)
at org.jboss.mq.SpySession.run(SpySession.java:333)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
at java.lang.Thread.run(Thread.java:810)
Caused by:
java.lang.IllegalStateException: Trying to change Tx in enlist!
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java
:417)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322
)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java
:583)
... 23 more

2. Threads that get created inside the application server and start their own independent transactions:

org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException f
or SQL [???]; SQL state [null]; error code [0]; Unchecked throwable in managedConnectionReconnected(); - nested throwable: (j
ava.lang.IllegalStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss.resource.JBossResourceExcepti
on: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to ch
ange Tx in enlist!)); nested exception is org.jboss.util.NestedSQLException: Unchecked throwable in managedConnectionReconnec
ted(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss.
resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.Illega
lStateException: Trying to change Tx in enlist!))
Caused by:
org.jboss.util.NestedSQLException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.Ille
galStateException: Trying to change Tx in enlist!); - nested throwable: (org.jboss.resource.JBossResourceException: Unchecked
throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in en
list!))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:107)
at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvid
er.java:81)
at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:144)
at org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:139)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1557)
at org.hibernate.loader.Loader.doQuery(Loader.java:683)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:246)
at org.hibernate.loader.Loader.doList(Loader.java:2223)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2114)
at org.hibernate.loader.Loader.list(Loader.java:2109)
at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:94)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1569)
at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:283)
at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper$1.doInHibernate(HibernateListCriteriaHelper
.java:222)
at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:372)
at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper.getListChunk(HibernateListCriteriaHelper.ja
va:205)
at com.manh.df.base.persistence.dao.hibernate.HibernateListCriteriaHelper.getListChunk(HibernateListCriteriaHelper.ja
va:66)
at com.manh.df.base.persistence.dao.hibernate.HibernateProductLocationDao.getProductLocationList(HibernateProductLoca
tionDao.java:195)
at com.manh.df.base.service.impl.ProductLocationServiceImpl.getProductLocationList(ProductLocationServiceImpl.java:14
5)
at sun.reflect.GeneratedMethodAccessor1323.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:
50)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy640.getProductLocationList(Unknown Source)
at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.getProductLocationList(AutoAssociationBatchProcessor.j
ava:1025)
at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.associateProductLocationList(AutoAssociationBatchProce
ssor.java:249)
at com.manh.df.base.service.impl.AutoAssociationBatchProcessor.processRequest(AutoAssociationBatchProcessor.java:136)
at com.manh.df.base.async.AsynchronousRequestSupport$1.run(AsynchronousRequestSupport.java:67)
at java.security.AccessController.doPrivileged(AccessController.java:219)
at javax.security.auth.Subject.doAs(Subject.java:495)
at com.manh.df.base.util.PrivilegedActionRunner.doPrivilegedAction(PrivilegedActionRunner.java:84)
at com.manh.df.base.util.PrivilegedActionRunner.doPrivilegedAction(PrivilegedActionRunner.java:45)
at com.manh.df.base.async.AsynchronousRequestSupport.run(AsynchronousRequestSupport.java:52)
Caused by:
org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.l
ang.IllegalStateException: Trying to change Tx in enlist!)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java
:101)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java
:588)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:450)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnecti
onManager2.java:874)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:103)
... 38 more
Caused by:
java.lang.IllegalStateException: Trying to change Tx in enlist!
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java
:417)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322
)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java
:583)
... 41 more

3. Errors in External multi threaded client module which makes remote calls to EJB services:

org.jboss.resource.JBossResourceException: Unchecked throwable in managedConnectionReconnected(); - nested throwable: (java.lang.IllegalStateException: Trying to change Tx in enlist!)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java:101)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:588)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508)
at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421)
at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:209)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invokeHome(StatelessSessionInstanceInterceptor.java:81)
at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90)
at org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:41)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:109)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:146)
at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:116)
at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
at org.jboss.ejb.SessionContainer.internalInvokeHome(SessionContainer.java:613)
at org.jboss.ejb.Container.invoke(Container.java:894)
at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406)
at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:309)
at sun.rmi.transport.Transport$1.run(Transport.java:168)
at java.security.AccessController.doPrivileged(AccessController.java:279)
at sun.rmi.transport.Transport.serviceCall(Transport.java:164)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)
at java.lang.Thread.run(Thread.java:810)
at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:263)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:239)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:141)
at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:118)
at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:227)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:167)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:55)
at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:169)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:86)
... 22 more
Caused by:
java.lang.IllegalStateException: Trying to change Tx in enlist!
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:417)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:322)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:583)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnect(BaseConnectionManager2.java:508)
at org.jboss.resource.connectionmanager.CachedConnectionManager.reconnect(CachedConnectionManager.java:421)
at org.jboss.resource.connectionmanager.CachedConnectionManager.pushMetaAwareObject(CachedConnectionManager.java:261)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:209)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invokeHome(StatelessSessionInstanceInterceptor.java:81)
at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:90)
at org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:41)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:109)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:335)
at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:146)
at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:116)
at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:121)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:93)
at org.jboss.ejb.SessionContainer.internalInvokeHome(SessionContainer.java:613)
at org.jboss.ejb.Container.invoke(Container.java:894)
at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:805)
at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:406)
at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:309)
at sun.rmi.transport.Transport$1.run(Transport.java:168)
at java.security.AccessController.doPrivileged(AccessController.java:279)
at sun.rmi.transport.Transport.serviceCall(Transport.java:164)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:506)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.handleRequest(TCPTransport.java:838)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:912)