Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-427

Huge socket leak when response not fully read by client

    XMLWordPrintable

Details

    Description

      When a client doesn't fully read the response, Undertow is leaking sockets when using the AJP connector. This will result in the process eventually running out of file descriptors and then crashing.

      Sockets will remain in a half closed status, and the OS will not clean them up.

      The issue could be reproduced using all version of WildFly from 8.0 to 9.0-beta2 and a nightly build from 24 April (which uses Undertow 1.2.1-CR1).

      The issue could however not be reproduced using JBoss EAP 6.x.

      The following is the minimal Apache config used to reproduce the issue:

      <VirtualHost *:80>
          ServerAdmin   info@example.com
          ServerName    localhost
      
          ProxyPass     /                    ajp://localhost:8009/
         
          CustomLog     /var/log/apache2/localhost-access.log custom
          ErrorLog      /var/log/apache2/localhost-error.log
      </Virtualhost>
      

      The following is a test case that can be used to reproduce the problem.

      Server

      package demo;
      
      import java.io.IOException;
      
      import javax.servlet.ServletException;
      import javax.servlet.annotation.WebServlet;
      import javax.servlet.http.HttpServlet;
      import javax.servlet.http.HttpServletRequest;
      import javax.servlet.http.HttpServletResponse;
      
      @WebServlet(urlPatterns = {"/*"})
      public class SocketLeakServlet extends HttpServlet {
      
          private static final long serialVersionUID = 0L;
          private static StringBuffer responseData = new StringBuffer("Z");
          private static long requestCounter = 0;
      
          @Override
          public void init() {
              // Binary increase responseData: 2^i [bytes] required to fine tune race condition,
              // start with ~ 32kB (2^15)
              for (int i = 0; i < 15; i++) {
                  responseData.append(responseData);
              }
      
              System.out.println("ResponseData: " + (responseData.length() / 1024) + "kB");
          }
      
          @Override
          protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
              // Delay response for 2mS in oder to prevent socket depletion (depending on OS) due to state TIME_WAIT.
              // OS used during tests: debian jessie with default out of the box settings
              try {
                  Thread.sleep(2);
              } catch (InterruptedException e) {
                  e.printStackTrace();
              }
      
              resp.setStatus(HttpServletResponse.SC_OK);
      
              requestCounter++;
      
              System.out.println("ClientRequests served: " + requestCounter);
      
              // Two methods are possible to write responseData to client: resp.getWriter() or resp.getOutputstream().
              // Both methods result in Half Closed Sockets that can be observed with following linux command:
              // watch "lsof -p <wildfly_pid> | grep sock | grep protocol | wc -l"
      
              resp.getWriter().print(responseData.toString());
              //resp.getOutputStream().print(responseData.toString());
          }
      
      }
      

      Client

      import static java.util.logging.Level.WARNING;
      
      import java.net.HttpURLConnection;
      import java.net.MalformedURLException;
      import java.net.URL;
      import java.security.GeneralSecurityException;
      import java.security.SecureRandom;
      import java.security.cert.X509Certificate;
      import java.util.logging.Logger;
      
      import javax.net.ssl.*;
      
      public class HttpConnectWithoutReadingBodyTest {
      
      	private static final Logger logger = Logger.getLogger(HttpConnectWithoutReadingBodyTest.class.getName());
      
      	// This configures to trust all SSL certificates, SSL hostnames and ignore SNI
      	static {
      		TrustManager[] trustAllCertificates = new TrustManager[]{new X509TrustManager() {
      			@Override
      			public X509Certificate[] getAcceptedIssuers() {
      				return null; // Not relevant.
      			}
      
      			@Override
      			public void checkClientTrusted(X509Certificate[] certs, String authType) {
      				// Do nothing. Just allow them all.
      			}
      
      			@Override
      			public void checkServerTrusted(X509Certificate[] certs, String authType) {
      				// Do nothing. Just allow them all.
      			}
      		}};
      
      		HostnameVerifier trustAllHostnames = new HostnameVerifier() {
      			@Override
      			public boolean verify(String hostname, SSLSession session) {
      				return true; // Just allow them all.
      			}
      		};
      
      		try {
      			System.setProperty("jsse.enableSNIExtension", "false");
      			SSLContext sc = SSLContext.getInstance("SSL");
      			sc.init(null, trustAllCertificates, new SecureRandom());
      			HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
      			HttpsURLConnection.setDefaultHostnameVerifier(trustAllHostnames);
      		}
      		catch (GeneralSecurityException e) {
      			logger.log(WARNING, "Failed to install 'trust all SSL certificates' manager", e);
      		}
      	}
      
      	public static void main(String[] args) {
      		if (args.length != 2) {
      			printUsage();
      			return;
      		}
      
      		try {
      			URL url = new URL(args[1]);
      			int numberOfAttempts = Integer.parseInt(args[0]);
      
      			for (int i = 0; i < numberOfAttempts; i++) {
      				HttpURLConnection connection = (HttpURLConnection) url.openConnection();
      
      				int responseCode;
      				try (AutoCloseable ac = connection::disconnect) {
      					responseCode = connection.getResponseCode();
      				}
      
      				System.out.println("Response received: " + responseCode);
      			}
      		}
      		catch(NumberFormatException | MalformedURLException e) {
      			printUsage();
      		}
      		catch (Exception e) {
      			// TODO Auto-generated catch block
      			e.printStackTrace();
      		}
      	}
      
      	private static void printUsage() {
      		System.out.println("Usage <number of attempts> <URL>");
      	}
      }
      

      Assuming the Servlet is deployed in an application called "socketleak", and the client code in a jar named "HttpConnectWithoutReadingBodyTest.jar", the following command can be used to start the test:

      /java -jar HttpConnectWithoutReadingBodyTest.jar 1000 http://localhost/socketleak
      

      While the test is running, the following command (Debian Linux) can be used to monitor the socket leak:

      watch "lsof -p <wildfly_pid> | grep sock | grep protocol | wc -l"
      

      Some observations:

      • Using the Servlet Writer the leaks occur faster and more frequently than when using the Servlet OutputStream. But eventually both leak.
      • The size of the response being written matters and seems to relate to the server's performance. Increasing the response from 4kb to 32kb causes more leaks, but going higher, e.g. 64kb of 128kb seems to cause sligtly less leaking
      • When using the http protocol no leaks occur, it only happens when using AJP
      • In real life the main trigger that causes the leaks are monitoring tools that don't read the full response, but the problem also occurs when a large page is loading and a user refreshes it while it's still loading

      Attachments

        Activity

          People

            sdouglas1@redhat.com Stuart Douglas
            arjant_jira Arjan t (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: