Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-3146

Undertow Memory Leak for JAX-RS requests

XMLWordPrintable

    • Hide

      Unzip the WildFly.8.0.0.Final distribution.
      Drop in the attached hello-service.war file into standalone-deployments.
      Add an https listener to the undertow service in standalone.xml:

                      <https-listener name="https" socket-binding="https" security-realm="SSLRealm"/>

      Add a new SSLRealm security domain to <security-realms> in standalone.xml:

                  <security-realm name="SSLRealm">
                <server-identities>
                    <ssl>
                        <keystore path="ssl/server.keystore" relative-to="jboss.server.data.dir" keystore-password="abc123" alias="serverkey"/>
                    </ssl>
                </server-identities>
            </security-realm>

      Mkdir ...standalone/data/ssl and copy the attached server.keystore file into it.
      Start WildFly with standalone.bat
      Attach VisualVM (or your favourite profiler) to the WildFly process
      Run hello-client's a.b.c.HelloClient main method - this uses 10 threads to send parallel requests to the server.

      Show
      Unzip the WildFly.8.0.0.Final distribution. Drop in the attached hello-service.war file into standalone-deployments. Add an https listener to the undertow service in standalone.xml: <https-listener name= "https" socket-binding= "https" security-realm= "SSLRealm" /> Add a new SSLRealm security domain to <security-realms> in standalone.xml: <security-realm name= "SSLRealm" > <server-identities> <ssl> <keystore path= "ssl/server.keystore" relative-to= "jboss.server.data.dir" keystore-password= "abc123" alias= "serverkey" /> </ssl> </server-identities> </security-realm> Mkdir ...standalone/data/ssl and copy the attached server.keystore file into it. Start WildFly with standalone.bat Attach VisualVM (or your favourite profiler) to the WildFly process Run hello-client's a.b.c.HelloClient main method - this uses 10 threads to send parallel requests to the server.

      After half an hour or so of running the automated test suite for our application, the server hangs due to GC and many OutOfMemoryErrors appear in the WildFly server log.

      I've attached VisualVM and JProfiler, both of which demonstrate that many of the objects that are not being garbage collected are Undertow objects. Both tools show that the path to the GC root is via a Java nio SelectionKey, which is held by sun.nio.ch.WindowsSelectorImpl.

      I'll attach a screenshot of a JProfiler window to demonstrate, but VisualVM shows the same result (albeit presented differently).

      I have also found similar behaviour with a small, trivial example, as described in the Steps to Reproduce. It makes repeated calls to a simple RESTful call to a JAX-RS service. The example doesn't cause as fast a leak as our test suite, but it does demonstrate the problem.

      The example uses https for the call because it seems to leak memory faster than http - but http does also exhibit the problem.

      I have also tried patching WildFly 8.0.0.Final up to Undertow.1.0.2-SNAPSHOT and XNIO.3.3.0-SNAPSHOT (as of this morning), but the problem still occurs.

        1. hello-client.zip
          5 kB
        2. hello-service.zip
          4 kB
        3. hello-service-0.1-SNAPSHOT.war
          4 kB
        4. JProfiler GC Root.png
          JProfiler GC Root.png
          190 kB
        5. server.keystore
          2 kB
        6. standalone.xml
          20 kB
        7. VisualVM Memory.png
          VisualVM Memory.png
          57 kB

              sdouglas1@redhat.com Stuart Douglas (Inactive)
              darrenjones_jira Darren Jones (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: