Details
-
Enhancement
-
Resolution: Done
-
Major
-
7.4.2.GA
Description
When starting several big JSF applications in EAP the memory floors the 100% and the server starts having a lot of full GCs. Then a lot of GC overhead problems occur, the boot is extremely slow and can throw OOM eventually.
Threads deploying the apps are always doing something similar to the following:
Thread at java.util.zip.ZipFile.getEntry(J[BZ)J (Native Method) at java.util.zip.ZipFile.getInputStream(Ljava/util/zip/ZipEntry;)Ljava/io/InputStream; (ZipFile.java:374) at java.util.jar.JarFile.getInputStream(Ljava/util/zip/ZipEntry;)Ljava/io/InputStream; (JarFile.java:467) at org.jboss.vfs.spi.JavaZipFileSystem.openInputStream(Lorg/jboss/vfs/VirtualFile;Lorg/jboss/vfs/VirtualFile;)Ljava/io/InputStream; (JavaZipFileSystem.java:182) at org.jboss.vfs.VirtualFile.openStream()Ljava/io/InputStream; (VirtualFile.java:318) at org.jboss.vfs.VirtualJarInputStream.openCurrent(Lorg/jboss/vfs/VirtualFile;)V (VirtualJarInputStream.java:223) at org.jboss.vfs.VirtualJarInputStream.getNextJarEntry()Ljava/util/jar/JarEntry; (VirtualJarInputStream.java:109) at org.jboss.vfs.VirtualJarInputStream.getNextEntry()Ljava/util/zip/ZipEntry; (VirtualJarInputStream.java:80) at com.sun.faces.facelets.util.Classpath.searchFromURL(Ljava/util/Set;Ljava/lang/String;Ljava/lang/String;Ljava/net/URL;)V (Classpath.java:175) at com.sun.faces.facelets.util.Classpath.searchFromURL(Ljava/util/Set;Ljava/lang/String;Ljava/lang/String;Ljava/net/URL;)V (Classpath.java:198) at com.sun.faces.facelets.util.Classpath.search(Ljava/lang/ClassLoader;Ljava/lang/String;Ljava/lang/String;Lcom/sun/faces/facelets/util/Classpath$SearchAdvice;)[Ljava/net/URL; (Classpath.java:114) at com.sun.faces.facelets.util.Classpath.search(Ljava/lang/String;Ljava/lang/String;)[Ljava/net/URL; (Classpath.java:68) at com.sun.faces.config.configprovider.MetaInfFacesConfigResourceProvider.loadURLs(Ljavax/servlet/ServletContext;)Ljava/util/Collection; (MetaInfFacesConfigResourceProvider.java:143) at com.sun.faces.config.configprovider.MetaInfFacesConfigResourceProvider.getResources(Ljavax/servlet/ServletContext;)Ljava/util/Collection; (MetaInfFacesConfigResourceProvider.java:87) at com.sun.faces.config.manager.tasks.FindConfigResourceURIsTask.call()Ljava/util/Collection; (FindConfigResourceURIsTask.java:72) at com.sun.faces.config.manager.tasks.FindConfigResourceURIsTask.call()Ljava/lang/Object; (FindConfigResourceURIsTask.java:40) at java.util.concurrent.FutureTask.run()V (FutureTask.java:266) at com.sun.faces.config.manager.Documents.getXMLDocuments(Ljavax/servlet/ServletContext;Ljava/util/List;Ljava/util/concurrent/ExecutorService;Z)[Lcom/sun/faces/config/manager/documents/DocumentInfo; (Documents.java:82) at com.sun.faces.config.ConfigManager.initialize(Ljavax/servlet/ServletContext;Lcom/sun/faces/config/InitFacesContext;)V (ConfigManager.java:266) at com.sun.faces.config.ConfigureListener.contextInitialized(Ljavax/servlet/ServletContextEvent;)V (ConfigureListener.java:205) at io.undertow.servlet.core.ApplicationListeners.contextInitialized()V (ApplicationListeners.java:187) ...
Since JSF 2.0 the spec allows a new configuration file with the name META-INF/*.faces-config.xml, which was added to allow several configuration files per jar. Therefore the implementation iterates the app jars looking for files inside the META-INF and ending with the suffix .faces-config.xml. It seems that VirtualJarInputStream consumes a lot of memory because it opens an InputStream for each file iterated inside the jar, even if it was not accessed at all. Each InputStream internally creates byte-arrays in the JDK that depletes the memory quickly. This effect is much worse in jdk-8 because ZipFileInflaterInputStream uses a finalize method that serializes the closing and memory takes much more time to be freed.
VirtualJarInputStream in jboss-vfs can be improved to delay the openStream call to the moment when the entry stream is really needed. That way no new streams are created if the jar is just iterated. This seems to be a common use-case that is improved a lot, the jar file is iterated and only the few entries in which the app is interested are really opened (in the JSF case, the configuration files).
Steps to reproduce the issue
- Start a jboss-eap-7.
- unzip the attached file inside the JBOSS_HOME/standalone/deployments folder:
tar xvf ~/Desktop/reproducer.tar.xz
- The ejb-in-war is an app exploded folder that contains a big jar to scan.
- Execute the deployments.sh N in the deployments folder while eap is running. It creates N deployments at the same time to force the issue.
./deployments 20
- See that startup is very slow and OOM is thrown eventually.
Attachments
Issue Links
- is incorporated by
-
JBEAP-22935 (7.4.z) Upgrade jboss-vfs from 3.2.15.Final-redhat-00001 to 3.2.16.Final-redhat-00001
-
- Closed
-
