Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-472

Very occasional sudden Java heap spike and JVM OOM

    XMLWordPrintable

Details

    Description

      We are running Undertow embedded nodes behind Amazon Web Services (AWS) Elastic Load Balancer (ELB) in EC2. We are using both the HTTP and HTTPS listener with a wildcard SSL certificate. ELB is configured for TCP/IP ports on 80 and 443. The Undertow nodes behind ELB have corresponding ports for HTTP and HTTPS. The API is terse JSON over websocket.

      On occasion, an Undertow node will suddenly spike from almost no heap usage to full heap and the JVM will OOM. From the stacktrace I have available, it seems like it gets caught in a loop where memory usage is continually doubled until the JVM goes OOM.

      I saw a heap histogram the first time it happened and saw hundreds of thousands of SslConduit classes in the heap – the #1 occupying item in the heap. Hopefully that might be of assistance.

      I apologize for the lack of additional detail as this has happened only twice thus far in a few months.

      Load is not an issue at the moment as the servers have mostly been idle or serving a few websocket connections (via AWS Elastic Load Balancer).

      Any ideas, recommendations on how to instrument so that I may provide better information the next time it happens would be appreciated.

      2015-06-13 18:24:30,055 [ERROR] [XNIO-2 I/O-2] org.xnio.nio - XNIO000011: Task io.undertow.protocols.ssl.SslConduit$1@7f58d82f failed with an exception
      java.lang.OutOfMemoryError: Java heap space
      at java.util.ArrayDeque.doubleCapacity(ArrayDeque.java:157) ~[na:1.8.0_45]
      at java.util.ArrayDeque.addLast(ArrayDeque.java:247) ~[na:1.8.0_45]
      at java.util.ArrayDeque.add(ArrayDeque.java:418) ~[na:1.8.0_45]
      at org.xnio.nio.WorkerThread.execute(WorkerThread.java:571) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit.runReadListener(SslConduit.java:221) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit.handleHandshakeResult(SslConduit.java:810) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:664) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:559) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit.access$500(SslConduit.java:63) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:978) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at io.undertow.protocols.ssl.SslConduit$1.run(SslConduit.java:224) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:560) ~[api-server-0.1.2-jar-with-dependencies.jar:na]
      at org.xnio.nio.WorkerThread.run(WorkerThread.java:462) ~[api-server-0.1.2-jar-with-dependencies.jar:na]

      Attachments

        Activity

          People

            sdouglas1@redhat.com Stuart Douglas
            chatbot2046 Y Pai (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: