Uploaded image for project: 'JBoss Portal'
  1. JBoss Portal
  2. JBPORTAL-2466

Native heap memory leak when rendering Portlet URLs causes JVM crash

    Details

      Description

      Under moderate load testing of a page containing thirtysome Portlet URLs, the JVM crashes quickly with various variants of the following stderr message:

      1. A fatal error has been detected by the Java Runtime Environment:
        #
      2. java.lang.OutOfMemoryError: requested 70968 bytes for Chunk::new. Out of swap space?
        #
      3. Internal Error (allocation.cpp:215), pid=24331, tid=1870162832
      4. Error: Chunk::new
        #
      5. JRE version: 6.0_16-b01
      6. Java VM: Java HotSpot(TM) Server VM (14.2-b01 mixed mode linux-x86 )
      7. An error report file with more information is saved as:
        ..

      Lowering the Java heap size to 1/4 of the original size avoided the problem (but was for other reasons not acceptable). The problem was thus avoided when garbage collection was more frequent. This behaviour, in combination with the error message above, indicated that this was in fact a native heap leak and that this native memory was being freed when the Java garbage collector cleaned up Java objects and triggered their finalizers.

      One common case of native heap leak problems is using the GZipOutputStream for compression. The GZipOutputStream is a thin wrapper for native ZLIB resources which consume a lot of native memory. Creating a lot of GZipOuputStream objects without closing them can thus crash the JVM since ZLIB fills upp the native heap before the wrapping GZipOutputStream objects fill upp the Java heap and trigger a garbage collection.

      Looking at the JBoss Portal sources, GzipOutputStream is used to compress parameters in Portlet URLs. After compression, GZipOutstream.finish() is invoked but never GZipOutstream.close(). This leaves the ZLIB native resources hanging and causes a native heap leak.

      I manually patched

      org.jboss.portal.common.io.SerializationFilter

      to force a GZipOutstream.close(). This solved the problem and the application now behaves nicely under load also with large heap settings. It would be great to have this included in an official patch by changing

      public <T> void serialize(Serialization<T> serialization, T t, OutputStream out) throws IllegalArgumentException, IOException

      { GZIPOutputStream zos = new GZIPOutputStream(out); serialization.serialize(t, zos); zos.finish(); }

      to

      public <T> void serialize(Serialization<T> serialization, T t, OutputStream out) throws IllegalArgumentException, IOException
      {
      GZIPOutputStream zos = null;
      try

      { zos = new GZIPOutputStream(out); serialization.serialize(t, zos); zos.finish(); }

      finally {
      if (zos != null)

      { zos.close(); }

      }
      }

      or similar.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                patrik.gottfridsson Patrik Gottfridsson
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: