Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-17194

Oracle UCP OutOfMemoryError: GC overhead limit exceeded after multiple server reloads

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • None
    • Testing/Validation
    • Hide

      EAP reproducer

      1. edit jboss-eap-7.2/bin/standalone.conf and change Xms and Xmx to 64m (it is reproducible with default, but it takes to much time)
      2. start server 
        jboss-eap-7.2/bin/standalone.sh -Doracle.ucp.destroyOnReload=true
      3. Create Oracle UCP datasource 
        module add --name=com.oracle.ucp --resources=/tmp/jbeap9382/ucp.jar:/tmp/jbeap9382/ojdbc8.jar:/tmp/jbeap9382/ons.jar --dependencies=javax.api,javax.transaction.api

/subsystem=datasources/jdbc-driver=oracle-ucp:add(driver-name=oracle-ucp, driver-module-name=com.oracle.ucp,driver-xa-datasource-class-name=oracle.ucp.jdbc.PoolXADataSourceImpl,driver-datasource-class-name=oracle.ucp.jdbc.PoolDataSourceImpl)

data-source add --name=mypool --jndi-name="java:/datasources/mypool" --connection-properties=InactiveConnectionTimeout=0,PropertyCycle=900,ValidateConnectionOnBorrow=true,MaxPoolSize=40,MinPoolSize=0,FastConnectionFailoverEnabled=true,AbandonedConnectionTimeout=0,URL="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=172.17.0.2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=mypdb)))",ConnectionPoolName=mypool,ConnectionWaitTimeout=0,TimeToLiveConnectionTimeout=0,ConnectionFactoryClassName=oracle.jdbc.pool.OracleDataSource,User=user1,Password=user1 --driver-name=oracle-ucp --min-pool-size=0 --max-pool-size=20 --allow-multiple-users=false --valid-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker --validate-on-match=true --use-fast-fail=true --stale-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker --exception-sorter-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter --user-name=user1 --password=user1
      4. run 
        i=0; while true; do i=$((i+1)); echo $i; /tmp/jboss-eap-7.2/bin/jboss-cli.sh -c --commands="/subsystem=datasources/data-source=mypool:test-connection-in-pool,:reload";  done

      Standalone reproducer

      import java.sql.Connection;
import java.sql.SQLException;

import javax.sql.DataSource;

import oracle.ucp.jdbc.PoolDataSource;
import oracle.ucp.jdbc.PoolDataSourceImpl;

public class JBEAP17194 {

    public static void main(String[] args) throws Exception {

        int i = 0;
        while (true) {
            System.out.println(i++);
            DataSource ds = getDataSource();
            Connection conn = null;
            try {
                conn = ds.getConnection("user1", "user1");
            } finally {
                if (conn != null)
                    conn.close();
                conn = null;
            }
            ds = null;
        }
    }

    public static PoolDataSource getDataSource() throws SQLException {
        PoolDataSourceImpl ds = new PoolDataSourceImpl();
        ds.setInactiveConnectionTimeout(0);
        ds.setPropertyCycle(900);
        ds.setValidateConnectionOnBorrow(true);
        ds.setMaxPoolSize(20);
        ds.setMinPoolSize(0);
        ds.setFastConnectionFailoverEnabled(true);
        ds.setAbandonedConnectionTimeout(0);
        ds.setURL("jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=172.17.0.2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=mypdb)))");
        ds.setConnectionPoolName("ucp-pool");
        ds.setConnectionFactoryClassName("oracle.jdbc.pool.OracleDataSource");
        ds.setConnectionWaitTimeout(0);
        ds.setTimeToLiveConnectionTimeout(0);
        ds.setUser("user1");
        ds.setPassword("user1");

        return ds;
    }
}

      javac -cp ucp.jar:ojdbc8.jar JBEAP17194.java
java -Xms32m -Xmx32m -cp .:ucp.jar:ojdbc8.jar:ons.jar -Doracle.ucp.destroyOnReload=true JBEAP17194
      Show
      EAP reproducer edit jboss-eap-7.2/bin/standalone.conf and change Xms and Xmx to 64m (it is reproducible with default, but it takes to much time) start server jboss-eap-7.2/bin/standalone.sh -Doracle.ucp.destroyOnReload= true Create Oracle UCP datasource module add --name=com.oracle.ucp --resources=/tmp/jbeap9382/ucp.jar:/tmp/jbeap9382/ojdbc8.jar:/tmp/jbeap9382/ons.jar --dependencies=javax.api,javax.transaction.api /subsystem=datasources/jdbc-driver=oracle-ucp:add(driver-name=oracle-ucp, driver-module-name=com.oracle.ucp,driver-xa-datasource-class-name=oracle.ucp.jdbc.PoolXADataSourceImpl,driver-datasource-class-name=oracle.ucp.jdbc.PoolDataSourceImpl) data-source add --name=mypool --jndi-name="java:/datasources/mypool" --connection-properties=InactiveConnectionTimeout=0,PropertyCycle=900,ValidateConnectionOnBorrow=true,MaxPoolSize=40,MinPoolSize=0,FastConnectionFailoverEnabled=true,AbandonedConnectionTimeout=0,URL="jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=172.17.0.2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=mypdb)))",ConnectionPoolName=mypool,ConnectionWaitTimeout=0,TimeToLiveConnectionTimeout=0,ConnectionFactoryClassName=oracle.jdbc.pool.OracleDataSource,User=user1,Password=user1 --driver-name=oracle-ucp --min-pool-size=0 --max-pool-size=20 --allow-multiple-users=false --valid-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker --validate-on-match=true --use-fast-fail=true --stale-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker --exception-sorter-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter --user-name=user1 --password=user1 run i=0; while true ; do i=$((i+1)); echo $i; /tmp/jboss-eap-7.2/bin/jboss-cli.sh -c --commands= "/subsystem=datasources/data-source=mypool:test-connection-in-pool,:reload" ; done Standalone reproducer import java.sql.Connection; import java.sql.SQLException; import javax.sql.DataSource; import oracle.ucp.jdbc.PoolDataSource; import oracle.ucp.jdbc.PoolDataSourceImpl; public class JBEAP17194 { public static void main( String [] args) throws Exception { int i = 0; while ( true ) { System .out.println(i++); DataSource ds = getDataSource(); Connection conn = null ; try { conn = ds.getConnection( "user1" , "user1" ); } finally { if (conn != null ) conn.close(); conn = null ; } ds = null ; } } public static PoolDataSource getDataSource() throws SQLException { PoolDataSourceImpl ds = new PoolDataSourceImpl(); ds.setInactiveConnectionTimeout(0); ds.setPropertyCycle(900); ds.setValidateConnectionOnBorrow( true ); ds.setMaxPoolSize(20); ds.setMinPoolSize(0); ds.setFastConnectionFailoverEnabled( true ); ds.setAbandonedConnectionTimeout(0); ds.setURL( "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=TCP)(HOST=172.17.0.2)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=mypdb)))" ); ds.setConnectionPoolName( "ucp-pool" ); ds.setConnectionFactoryClassName( "oracle.jdbc.pool.OracleDataSource" ); ds.setConnectionWaitTimeout(0); ds.setTimeToLiveConnectionTimeout(0); ds.setUser( "user1" ); ds.setPassword( "user1" ); return ds; } } javac -cp ucp.jar:ojdbc8.jar JBEAP17194.java java -Xms32m -Xmx32m -cp .:ucp.jar:ojdbc8.jar:ons.jar -Doracle.ucp.destroyOnReload= true JBEAP17194

    Description

      In tests which in loop do:

      1. test datasource (DataSource.getconnection, Connection.close)
      2. reload server

      I see:

      12:22:14,223 ERROR [org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer] (MSC service thread 1-3) Error during the deployment of java:jboss/datasources/ExampleDS: java.lang.OutOfMemoryError: GC overhead limit exceeded

12:22:14,605 ERROR [org.jboss.msc] (MSC service thread 1-1) MSC000008: Worker thread Thread[MSC service thread 1-1,5,main] threw an uncaught exception: java.lang.OutOfMemoryError: GC overhead limit exceeded

12:22:14,975 WARN  [org.jboss.msc.service] (MSC service thread 1-6) MSC000007: An internal service error has occurred while processing an operation on service org.wildfly.clustering.infinispan.cache-container-configuration.hibernate.thread-pool.state-transfer: java.lang.OutOfMemoryError: GC overhead limit exceeded

12:22:14,445 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service org.wildfly.data-source.mypool: org.jboss.msc.service.StartException in service org.wildfly.data-source.mypool: WFLYJCA0033: Error during the deployment of mypool
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.start(AbstractDataSourceService.java:175)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1738)
        at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1700)
        at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1558)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.jboss.jca.deployers.common.DeployException: IJ020056: Deployment failed: file://DataSourceDeployment
        at org.jboss.jca.deployers.common.AbstractDsDeployer.createObjectsAndInjectValue(AbstractDsDeployer.java:425)
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$AS7DataSourceDeployer.deploy(AbstractDataSourceService.java:424)
        at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService.start(AbstractDataSourceService.java:161)
        ... 8 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

      Attachment dumps.zip contain heap dump before test and after OOM. It's from different test than in "Steps to Reproduce" but it does the same. Test has EJB which gets and closes connection.

      Attachments

        Issue Links

          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-3274 (7.2.0) Leaking threads, memory and physical connections from Oracle UCP

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              msimka@redhat.com Martin Simka
              msimka@redhat.com Martin Simka
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: