Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-22449

[GSS](7.3.z) JBJCA-1429 - Connection leak following transaction timeout during XAResource enlistment

    XMLWordPrintable

Details

    Description

      A transaction timeout during XA connection enlistment results in a connection leak. The behavior is verifiable with with CP6 - CP8 and also with the 7.4 GA but it is not reproducible with 7.3 as late as CP5.

      The scenario is this ...

      1. Start a transaction with a 2 second transaction timeout.

      2. Request a connection from an XA pool (reproduced with both Oracle and H2) using a Byteman rule to inject a sleep during the "enlistment" ...

      # Rule for H2
      RULE org.h2.jdbcx.JdbcXAConnection.start
      CLASS ^org.h2.jdbcx.JdbcXAConnection
      METHOD start
      IF true
      DO traceStack("[BMAN] sleeping during XAResource.start ...", 10);
         Thread.sleep(5000);
      ENDRULE
      
      # Rule for Oracle
      RULE oracle.jdbc.xa.client.OracleXAResource.start
      CLASS oracle.jdbc.xa.client.OracleXAResource
      METHOD start
      IF true
      DO traceStack("[BMAN] sleeping during XAResource.start ...", 10);
         Thread.sleep(5000);
      ENDRULE
      

      This results in a timeout (during the sleep). The application code subsequently closes the connection (implicitly using try-with-resources though explicit closure also tested and behavior is the same).

      I replaced the ExampleDS with the below to convert to an XA pool

                      <xa-datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
                          <xa-datasource-property name="URL">
                              jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
                          </xa-datasource-property>
                          <driver>h2</driver>
                          <security>
                              <user-name>sa</user-name>
                              <password>sa</password>
                          </security>
                      </xa-datasource>
      

      3. Rollback the failed transaction.

      Subsequently, statistics show a connection in-use (e.g. during a getConnection the stats show this in the log). And, during shutdown I see

      ... IJ000615: Destroying active connection in pool: ExampleDS (org.jboss.jca.adapters.jdbc.xa.XAManagedConnection@74a2f7bb)
      

      The difference in 7.3 CP5 and earlier seems to be that enlistment actually fails after the XAResource.start but before the connection is returned

      ... javax.transaction.SystemException: Error enlisting resource in transaction=Local transaction (delegate=TransactionImple < ... status: ActionStatus.ABORTED >, owner=Local transaction context for provider JBoss JTA transaction provider)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.checkEnlisted(TxConnectionListener.java:957)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:394)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.managedConnectionReconnected(TxConnectionManagerImpl.java:564)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:977)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
              at org.jboss.ironjacamar.jdbcadapters@1.4.22.Final-redhat-00001//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
              at org.jboss.as.connector@7.3.5.GA-redhat-00001//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
      ...
      Caused by: java.lang.Throwable: Failed to enlist
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.listener.TxConnectionListener$TransactionSynchronization.enlist(TxConnectionListener.java:1001)
              at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.enlist(TxConnectionListener.java:379)
      

      And, because of the enlistment failure, the connection is returned to the pool with a stack like this:

      2021-08-20 14:39:03,793 INFO  [stdout] (ServerService Thread Pool -- 78) [BMAN] org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.returnConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:-1)
      ... org.jboss.jca.core.connectionmanager.pool.AbstractPool.returnConnection(AbstractPool.java:847)
      ... org.jboss.jca.core.connectionmanager.AbstractConnectionManager.returnManagedConnection(AbstractConnectionManager.java:725)
      ... org.jboss.jca.core.connectionmanager.AbstractConnectionManager.managedConnectionDisconnected(AbstractConnectionManager.java:1113)
      ... org.jboss.jca.core.connectionmanager.AbstractConnectionManager.disconnectManagedConnection(AbstractConnectionManager.java:1069)
      ... org.jboss.jca.core.connectionmanager.AbstractConnectionManager.reconnectManagedConnection(AbstractConnectionManager.java:1046)
      ... org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:792)
      ... org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
      ... org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
      

      The enlistment failure does not occur with newer releases but it looks like (based on Byteman rules) enlist occurs (without failure) after the timeout with 7.3 CP6 and later.

      Attachments

        Issue Links

          Activity

            People

              tadamski@redhat.com Tomasz Adamski
              rhn-support-ivassile Ilia Vassilev
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: