Uploaded image for project: 'JBoss Web Server'
  1. JBoss Web Server
  2. JWS-3699

[ASF BZ 69866] OpenSSLContexts leak across reload through JNI global reference when using truststore

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 6.2.0.GA
    • 6.1.2.GA
    • tomcat, tomcat10
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • Hide

      1. Run tomcat with the attached conf content
      2. Run the following to reload, which leaks an OpenSSLContext each time:

      curl -v -u admin:redhat "http://localhost:8080/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfigs"

      3. Check the hisgtogram for OpenSSLContext instance counts and see that is increasing and not released at all even after forced GCs:

      $ jcmd 2001239 GC.class_histogram | grep OpenSSLContext
 254:            22           1056  org.apache.tomcat.util.net.openssl.OpenSSLContext
 282:            22            880  org.apache.tomcat.util.net.openssl.OpenSSLContext$OpenSSLState
 444:            22            352  org.apache.tomcat.util.net.openssl.OpenSSLContext$1
$ jcmd 2001239 GC.run
2001239:
Command executed successfully
$ jcmd 2001239 GC.class_histogram | grep OpenSSLContext
 254:            22           1056  org.apache.tomcat.util.net.openssl.OpenSSLContext
 282:            22            880  org.apache.tomcat.util.net.openssl.OpenSSLContext$OpenSSLState
 444:            22            352  org.apache.tomcat.util.net.openssl.OpenSSLContext$1
      Show
      1. Run tomcat with the attached conf content 2. Run the following to reload, which leaks an OpenSSLContext each time: curl -v -u admin:redhat "http: //localhost:8080/manager/jmxproxy?invoke=Catalina:type=ProtocolHandler,port=8443&op=reloadSslHostConfigs" 3. Check the hisgtogram for OpenSSLContext instance counts and see that is increasing and not released at all even after forced GCs: $ jcmd 2001239 GC.class_histogram | grep OpenSSLContext  254:            22           1056  org.apache.tomcat.util.net.openssl.OpenSSLContext  282:            22            880  org.apache.tomcat.util.net.openssl.OpenSSLContext$OpenSSLState  444:            22            352  org.apache.tomcat.util.net.openssl.OpenSSLContext$1 $ jcmd 2001239 GC.run 2001239: Command executed successfully $ jcmd 2001239 GC.class_histogram | grep OpenSSLContext  254:            22           1056  org.apache.tomcat.util.net.openssl.OpenSSLContext  282:            22            880  org.apache.tomcat.util.net.openssl.OpenSSLContext$OpenSSLState  444:            22            352  org.apache.tomcat.util.net.openssl.OpenSSLContext$1
    • Important

      https://bz.apache.org/bugzilla/show_bug.cgi?id=69866

      When using a truststore with the openssl provider, https://github.com/apache/tomcat/blob/10.1.36/java/org/apache/tomcat/util/net/openssl/OpenSSLContext.java#L362 creates a CertificateVerifier (OpenSSLContext$1) and stores it as a global JNI reference.  The created CertificateVerifier inner class keeps a strong reference back to the OpenSSLContext itself so this JNI global becomes a leaked reference chain that persists the OpenSSLContext and prevents it from becoming unreachable to prompt execution of its cleanable.  That reference chain in a heap dump:

       Class Name                                                                  | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------
org.apache.tomcat.util.net.openssl.OpenSSLContext$1 @ 0x42b04b308 JNI Global|            1 |           16 |                48 |            16
'- this$0 org.apache.tomcat.util.net.openssl.OpenSSLContext @ 0x42b056c50   |            1 |           48 |                48 |         1,200
----------------------------------------------------------------------------------------------------------------------------------------------
 

        1. JWS-3699reproducer.zip
          9 kB

              dsoumis Dimitris Soumis
              rhn-support-aogburn Aaron Ogburn
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: