When a client doesn't fully read the response, Undertow is leaking sockets when using the AJP connector. This will result in the process eventually running out of file descriptors and then crashing.
Sockets will remain in a half closed status, and the OS will not clean them up.
The issue could be reproduced using all version of WildFly from 8.0 to 9.0-beta2 and a nightly build from 24 April (which uses Undertow 1.2.1-CR1).
The issue could however not be reproduced using JBoss EAP 6.x.
The following is the minimal Apache config used to reproduce the issue:
<VirtualHost *:80> ServerAdmin info@example.com ServerName localhost ProxyPass / ajp://localhost:8009/ CustomLog /var/log/apache2/localhost-access.log custom ErrorLog /var/log/apache2/localhost-error.log </Virtualhost>
The following is a test case that can be used to reproduce the problem.
Server
package demo; import java.io.IOException; import javax.servlet.ServletException; import javax.servlet.annotation.WebServlet; import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; @WebServlet(urlPatterns = {"/*"}) public class SocketLeakServlet extends HttpServlet { private static final long serialVersionUID = 0L; private static StringBuffer responseData = new StringBuffer("Z"); private static long requestCounter = 0; @Override public void init() { // Binary increase responseData: 2^i [bytes] required to fine tune race condition, // start with ~ 32kB (2^15) for (int i = 0; i < 15; i++) { responseData.append(responseData); } System.out.println("ResponseData: " + (responseData.length() / 1024) + "kB"); } @Override protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException { // Delay response for 2mS in oder to prevent socket depletion (depending on OS) due to state TIME_WAIT. // OS used during tests: debian jessie with default out of the box settings try { Thread.sleep(2); } catch (InterruptedException e) { e.printStackTrace(); } resp.setStatus(HttpServletResponse.SC_OK); requestCounter++; System.out.println("ClientRequests served: " + requestCounter); // Two methods are possible to write responseData to client: resp.getWriter() or resp.getOutputstream(). // Both methods result in Half Closed Sockets that can be observed with following linux command: // watch "lsof -p <wildfly_pid> | grep sock | grep protocol | wc -l" resp.getWriter().print(responseData.toString()); //resp.getOutputStream().print(responseData.toString()); } }
Client
import static java.util.logging.Level.WARNING; import java.net.HttpURLConnection; import java.net.MalformedURLException; import java.net.URL; import java.security.GeneralSecurityException; import java.security.SecureRandom; import java.security.cert.X509Certificate; import java.util.logging.Logger; import javax.net.ssl.*; public class HttpConnectWithoutReadingBodyTest { private static final Logger logger = Logger.getLogger(HttpConnectWithoutReadingBodyTest.class.getName()); // This configures to trust all SSL certificates, SSL hostnames and ignore SNI static { TrustManager[] trustAllCertificates = new TrustManager[]{new X509TrustManager() { @Override public X509Certificate[] getAcceptedIssuers() { return null; // Not relevant. } @Override public void checkClientTrusted(X509Certificate[] certs, String authType) { // Do nothing. Just allow them all. } @Override public void checkServerTrusted(X509Certificate[] certs, String authType) { // Do nothing. Just allow them all. } }}; HostnameVerifier trustAllHostnames = new HostnameVerifier() { @Override public boolean verify(String hostname, SSLSession session) { return true; // Just allow them all. } }; try { System.setProperty("jsse.enableSNIExtension", "false"); SSLContext sc = SSLContext.getInstance("SSL"); sc.init(null, trustAllCertificates, new SecureRandom()); HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory()); HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames); } catch (GeneralSecurityException e) { logger.log(WARNING, "Failed to install 'trust all SSL certificates' manager", e); } } public static void main(String[] args) { if (args.length != 2) { printUsage(); return; } try { URL url = new URL(args[1]); int numberOfAttempts = Integer.parseInt(args[0]); for (int i = 0; i < numberOfAttempts; i++) { HttpURLConnection connection = (HttpURLConnection) url.openConnection(); int responseCode; try (AutoCloseable ac = connection::disconnect) { responseCode = connection.getResponseCode(); } System.out.println("Response received: " + responseCode); } } catch(NumberFormatException | MalformedURLException e) { printUsage(); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } } private static void printUsage() { System.out.println("Usage <number of attempts> <URL>"); } }
Assuming the Servlet is deployed in an application called "socketleak", and the client code in a jar named "HttpConnectWithoutReadingBodyTest.jar", the following command can be used to start the test:
/java -jar HttpConnectWithoutReadingBodyTest.jar 1000 http://localhost/socketleak
While the test is running, the following command (Debian Linux) can be used to monitor the socket leak:
watch "lsof -p <wildfly_pid> | grep sock | grep protocol | wc -l"
Some observations:
- Using the Servlet Writer the leaks occur faster and more frequently than when using the Servlet OutputStream. But eventually both leak.
- The size of the response being written matters and seems to relate to the server's performance. Increasing the response from 4kb to 32kb causes more leaks, but going higher, e.g. 64kb of 128kb seems to cause sligtly less leaking
- When using the http protocol no leaks occur, it only happens when using AJP
- In real life the main trigger that causes the leaks are monitoring tools that don't read the full response, but the problem also occurs when a large page is loading and a user refreshes it while it's still loading