Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-31445

[8.1.z] Commons IO NIO File use can lead to larger memory use in EAP 8.0.0+

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 8.1 Update 4
    • None
    • REST
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      A customer has previously been using EAP 7.4.21 and earlier without issue but they see their JBoss process size can grow up to another 1.5 gb until the process is OOM killed if they update to EAP 7.4.22+ or 8.0.0+.  The increased use is from more DirectByteBuffer overhead that was being cached in heap by the JDK's thread local sun.nio.ch.Util$BufferCache so it was apparent that EAP after updating is calling sun.nio.ch.Util.getTemporaryDirectBuffer in a way it hadn't before:

      Class Name                                                                                                                        | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.jboss.as.ee.concurrent.ManagedThreadFactoryImpl$ManagedThread @ 0x787bb4720  EE-ManagedExecutorService-default-Thread-3 Thread|            1 |          152 |                64 |         9,048
'- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x787d1d440                                                                |            1 |           24 |                64 |         8,448
   '- table java.lang.ThreadLocal$ThreadLocalMap$Entry[64] @ 0x787d1d458                                                          |            1 |          272 |                64 |         8,424
      '- [3] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x787b73c28                                                             |            1 |           32 |                64 |         4,168
         '- value sun.nio.ch.Util$BufferCache @ 0x787b73c48                                                                       |            1 |           24 |                64 |         4,136
            '- buffers java.nio.ByteBuffer[1024] @ 0x787b73c60                                                                    |            1 |        4,112 |                64 |         4,112
               '- [1023] java.nio.DirectByteBuffer @ 0x7885d6b88                                                                  |            1 |           64 |                64 |            64
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

      From some additional byteman debugging and log comparisons of their tests on 7.4.21 and 7.4.22, I identified and confirmed that only in 7.422+ large request writes through the RESTEasy client is calling this through commons io:

          at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
    at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:71)
    at java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280)
    at java.base/java.nio.channels.Channels.writeFullyImpl(Channels.java:74)
    at java.base/java.nio.channels.Channels.writeFully(Channels.java:97)
    at java.base/java.nio.channels.Channels$1.write(Channels.java:172)
    at java.base/java.io.OutputStream.write(OutputStream.java:122)
    at org.apache.commons.io@2.16.1.redhat-00001//org.apache.commons.io.output.ThresholdingOutputStream.write(ThresholdingOutputStream.java:235)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.util.DelegatingOutputStream.write(DelegatingOutputStream.java:42)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.util.DelegatingOutputStream.write(DelegatingOutputStream.java:42)
    at org.jboss.resteasy.resteasy-json-binding-provider@3.15.10.Final-redhat-00001//org.jboss.resteasy.plugins.providers.jsonb.JsonBindingProvider.writeTo(JsonBindingProvider.java:156)
    at org.jboss.resteasy.resteasy-json-binding-provider@3.15.10.Final-redhat-00001//org.jboss.resteasy.plugins.providers.jsonb.JsonBindingProvider.writeTo(JsonBindingProvider.java:160)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.writeTo(AbstractWriterInterceptorContext.java:137)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:124)
    at org.jboss.resteasy.resteasy-crypto@3.15.10.Final-redhat-00001//org.jboss.resteasy.security.doseta.DigitalSigningInterceptor.aroundWriteTo(DigitalSigningInterceptor.java:147)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.core.interception.AbstractWriterInterceptorContext.proceed(AbstractWriterInterceptorContext.java:129)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.writeRequestBody(ClientInvocation.java:420)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.writeRequestBodyToOutputStream(ApacheHttpClient4Engine.java:691)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.buildEntity(ApacheHttpClient4Engine.java:652)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.loadHttpMethod(ApacheHttpClient4Engine.java:530)
    at org.jboss.resteasy.resteasy-jaxrs@3.15.10.Final-redhat-00001//org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:329)

      RESTEasy is using a org.apache.commons.io.output.DeferredFileOutputStream specifically extending that ThresholdingOutputStream.  The customer has some requests up to 30-50 mbs and it seems RESTEasy/Commons IO is writing that in full at once to the underlying stream and thus allocating such a large buffer to match through the resulting sun.nio.ch.Util.getTemporaryDirectBuffer call.

       

      So this starts to occur from https://github.com/apache/commons-io/commit/b2811e866394085937c910c084ea37edde53ec35 changing the DeferredFileOutputStream to use an NIO File in 7.4.22+ now using apache commons io 2.16.1 and EAP 8.0+ using apache commons io 2.11.0.  This simple change causes a rather drastic difference as in a case like this with a difference of whether or not 30-50 mb DirectByteBuffers are being allocated for the same operations (and then also persistently cached by the JDK on its thread locals).

      Anything we would consider improving through the JBoss/RESTEasy/Commons IO layers to try to avoid such large DirectByteBuffer use from this change?  Some ideas I see so far:

       

      1. Do nothing at this layer and push customers to accept the increased overhead and manage the persistent JDK thread local growth by setting -Djdk.nio.maxCachedBufferSize as they see fit.  This may not be ideal or acceptable in the end as the concurrent temporary additional byte buffer overhead from a couple of 30-50 mb requests at once may still be too much.
      2. Try to push for a change in commons io or RESTEasy to do a large write across smaller NIO File write portions for a smaller peak Util.getTemporaryDirectBuffer allocation.
      3. Or consider adding another DeferredFileOutputStream implementation for RESTEasy to use that is still leveraging the prior java.io.File to avoid the NIO DirectByteBuffer use.

              Unassigned Unassigned
              jperkins-rhn James Perkins
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: