Can't enlist in transaction with second Oracle XA connection/resource.

We are experiencing the following exception chain when attempting to enlist a secound XA data source into a transaction. This only occurs on Oracle and only on JBoss 5.0.1 GA. We run the same software against JBoss 4.0.5, WebLogic and WebSphere with SQLServer and DB2 databases as well. Only this configuration gives us issues - SQLServer and DB2 behave as expected. In addition, transactions against a single XA connection work fine.

At the point we go to obtain a connection on the second XA data source, we get the following (commonly posted) exception information:

2009-07-08 14:19:51,484 WARN [org.jboss.resource.connectionmanager.TxConnectionManager] (http-127.0.0.1-8080-1) Connection error occured: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@4cde4cde[state=NORMAL mc=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@4c7c4c7c handles=0 lastUse=1247087991484 permit=true trackByTx=true mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool@39e939e9 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@5c7c5c7c xaResource=org.jboss.resource.adapter.jdbc.xa.XAManagedConnection@4c7c4c7c txSync=null]
oracle.jdbc.xa.OracleXAException
at oracle.jdbc.xa.OracleXAResource.checkError(OracleXAResource.java:1600)
at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:351)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:213)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:889)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:500)
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:908)
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:662)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:403)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
<deleted due to 32K requirement>

2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) java.lang.IllegalMonitorStateException
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:138)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1186)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:442)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.unlock(BaseWrapperManagedConnection.java:277)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.start(XAManagedConnection.java:235)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:889)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:500)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.enlist(TxConnectionManager.java:908)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:662)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:403)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499)
2009-07-08 14:19:51,515 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.context.ServerCallContext$GetConnection.run(ServerCallContext.java:1430)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.context.ServerCallContext.getConnectionWithRetry(ServerCallContext.java:917)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.context.ServerCallContext.getConnection(ServerCallContext.java:895)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.context.ServerCallContext._getDBContext(ServerCallContext.java:653)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.context.ServerCallContext.getDBContextFromJNDIValues(ServerCallContext.java:826)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.dbpersist.DBContext.determineDatabaseType(DBContext.java:661)
2009-07-08 14:19:51,531 ERROR [STDERR] (http-127.0.0.1-8080-1) at com.filenet.engine.persist.ObjectStorePersister.preExecuteChange(ObjectStorePersister.java:287)
<deleted due to 32K requirement>

2009-07-08 14:19:51,578 ERROR [org.jboss.ejb.plugins.LogInterceptor] (http-127.0.0.1-8080-1) RuntimeException in method: public abstract com.filenet.apiimpl.transport.ExecuteChangesResponse com.filenet.engine.ejb.EngineCoreLocal.executeChanges(com.filenet.apiimpl.transport.ClientCallContext,com.filenet.apiimpl.transport.ExecuteChangesRequest):
com.filenet.api.exception.EngineRuntimeException: E_FAILED_TO_GET_DATASOURCE: Failed to get DataSource OraDSXA.
at com.filenet.engine.context.ServerCallContext.getConnectionWithRetry(ServerCallContext.java:942)
at com.filenet.engine.context.ServerCallContext.getConnection(ServerCallContext.java:895)
at com.filenet.engine.context.ServerCallContext._getDBContext(ServerCallContext.java:653)
at com.filenet.engine.context.ServerCallContext.getDBContextFromJNDIValues(ServerCallContext.java:826)
at com.filenet.engine.dbpersist.DBContext.determineDatabaseType(DBContext.java:661)
at com.filenet.engine.persist.ObjectStorePersister.preExecuteChange(ObjectStorePersister.java:287)
at com.filenet.engine.persist.BaseGCDPersister.executeChangeWork(BaseGCDPersister.java:244)
at com.filenet.engine.persist.IndependentPersister.executeChange(IndependentPersister.java:235)
at com.filenet.engine.persist.BaseGCDPersister.executeChange(BaseGCDPersister.java:153)
at com.filenet.engine.jca.impl.RequestBrokerImpl.executeChanges(RequestBrokerImpl.java:1244)
at com.filenet.engine.jca.impl.RequestBrokerImpl.executeChanges(RequestBrokerImpl.java:1126)
at com.filenet.engine.ejb.EngineCoreBean._executeChanges(EngineCoreBean.java:601)
at com.filenet.engine.ejb.EngineCoreBean.executeChanges(EngineCoreBean.java:567)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:404)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:267)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:134)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:81)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
at org.jboss.ejb.Container.invoke(Container.java:1029)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:436)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy134.executeChanges(Unknown Source)
at com.filenet.engine.ejb.EngineBean.executeChanges(EngineBean.java:658)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:45)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:599)
at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:228)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:173)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:378)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.SecurityInterceptor.process(SecurityInterceptor.java:267)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:211)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.process(PreSecurityInterceptor.java:134)
at org.jboss.ejb.plugins.security.PreSecurityInterceptor.invoke(PreSecurityInterceptor.java:81)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:650)
at org.jboss.ejb.Container.invoke(Container.java:1029)
at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:436)
at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
at $Proxy135.executeChanges(Unknown Source)
at com.filenet.apiimpl.transport.ejb.EnginePortLocal.executeChanges(EnginePortLocal.java:43)
at com.filenet.apiimpl.wsi.ServiceSessionNst$1.run(ServiceSessionNst.java:1110)
at java.security.AccessController.doPrivileged(AccessController.java:284)
at javax.security.auth.Subject.doAs(Subject.java:573)
at com.filenet.apiimpl.authentication.util.J2EEAuthnUtil.runAs(J2EEAuthnUtil.java:533)
at com.filenet.apiimpl.authentication.util.J2EEAuthnUtilJB.runAs(J2EEAuthnUtilJB.java:280)
at com.filenet.apiimpl.util.J2EEUtilJB.doAs(J2EEUtilJB.java:103)
at com.filenet.apiimpl.wsi.ServiceSessionNst.sMakeInternalEJBCall(ServiceSessionNst.java:977)
at com.filenet.apiimpl.wsi.ServiceSessionNst.sHandleIncomingRequest(ServiceSessionNst.java:915)
at com.filenet.engine.wsi.ListenerNst.service(ListenerNst.java:127)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:735)
Caused by:
org.jboss.util.NestedSQLException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 9318dfb:dc6:4a550cf3:67 status: ActionStatus.ABORT_ONLY >); - nested throwable: (org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 9318dfb:dc6:4a550cf3:67 status: ActionStatus.ABORT_ONLY >))
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:95)
at com.filenet.engine.context.ServerCallContext$GetConnection.run(ServerCallContext.java:1430)
at com.filenet.engine.context.ServerCallContext.getConnectionWithRetry(ServerCallContext.java:917)
... 90 more
Caused by:
org.jboss.resource.JBossResourceException: Could not enlist in transaction on entering meta-aware object!; - nested throwable: (javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 9318dfb:dc6:4a550cf3:67 status: ActionStatus.ABORT_ONLY >)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:409)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.reconnectManagedConnection(BaseConnectionManager2.java:625)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:499)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
... 92 more
Caused by:
javax.transaction.SystemException: java.lang.Throwable: Unabled to enlist resource, see the previous warnings. tx=TransactionImple < ac, BasicAction: 9318dfb:dc6:4a550cf3:67 status: ActionStatus.ABORT_ONLY >
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener$TransactionSynchronization.checkEnlisted(TxConnectionManager.java:879)
at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.enlist(TxConnectionManager.java:675)
at org.jboss.resource.connectionmanager.TxConnectionManager.managedConnectionReconnected(TxConnectionManager.java:403)
... 96 more

The two XA data sources are configured as follows:

<datasources>
<xa-datasource>
<jndi-name>OraDSXA</jndi-name>
<use-java-context>false</use-java-context>
<set-tx-query-timeout/>
<track-connection-by-tx/>
<isSameRM-override-value>false</isSameRM-override-value>
<xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
<xa-datasource-property name="URL">jdbc:oracle:thin:@myserver:1522:SONATADB</xa-datasource-property>
<xa-datasource-property name="User">OS1</xa-datasource-property>
<xa-datasource-property name="Password">xxxx</xa-datasource-property>
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<no-tx-separate-pools/>
<metadata>
<!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
<type-mapping>Oracle9i</type-mapping>
</metadata>
</xa-datasource>
</datasources>

<datasources>
<xa-datasource>
<jndi-name>OraGCDDSXA</jndi-name>
<use-java-context>false</use-java-context>
<set-tx-query-timeout/>
<track-connection-by-tx/>
<isSameRM-override-value>false</isSameRM-override-value>
<xa-datasource-class>oracle.jdbc.xa.client.OracleXADataSource</xa-datasource-class>
<xa-datasource-property name="URL">jdbc:oracle:thin:@myserver:1522:SONATADB</xa-datasource-property>
<xa-datasource-property name="User">GCD</xa-datasource-property>
<xa-datasource-property name="Password">xxxx</xa-datasource-property>
<exception-sorter-class-name>org.jboss.resource.adapter.jdbc.vendor.OracleExceptionSorter</exception-sorter-class-name>
<no-tx-separate-pools/>
<metadata>
<!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
<type-mapping>Oracle9i</type-mapping>
</metadata>
</xa-datasource>
</datasources>

We've tried a number of things (various JDBC driver versions (11.1.0.7, 10.2.0.4), combining data source stanzas into single files, changing the type-mappings, toggling various setting via JMX console, reading the FAQs, google hits, Oracle metalink, etc.) and have not found anything that works. Given that this same database works fine when referenced from JB 4.0.5 (or any other application server) and that one difference between 4.0.5 and 5.0.1 is the transaction manager, we believe you are our last resource to help resolve this critical issue. We believe there's some configurable item - unfamiliar to us and, likely, not well-documented, that we must be missing, but don't know where to look next. Any direction on this matter would be greatly appreciated. We are happy to provide you with any information you need.

Kind regards.