Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-609

Undertow-core 1.3.11 introduced a commit that causes HTTP Keep-Alive requests to hang

XMLWordPrintable

    • Hide

      I added an export of my test code. In this zip are a few folders, namely:

      • servers: The Wildfly 10 CR5 release I tested against.
      • modules: The manually compiled undertow-core versions where eb1ca6b works, but 8c8e16c does not.
      • eclipse: An eclipse workspace of several modules, which include an ear that can be deployed, and a client which runs a test against the deployed ear.
      • artifacts: the build ear, and client jar.

      To reproduce, simply deploy the ear on the Wildfly server. Start the jar, and it will keep hanging after a few minutes. When you create a thread dump of the client you will see that it hangs on socketRead0. When you deploy the "eb1ca6b" undertow-core module it will not hang.

      I do not seem to be able to upload the testcode. Jira keeps saying "JIRA could not attach the file as there was a missing token. Please try attaching the file again." I did try different browsers, each of them showed this.

      The basic idea is to create a connection pool with keep alive enabled, and then continuously call a endpoint on the Wildfly server. You will see that at some point the client side keeps hanging, and that this is undertow-core version related.

      I'll check in tomorrow to check if I'm able to upload by then.

      I just uploaded the export.zip attachment

      Show
      I added an export of my test code. In this zip are a few folders, namely: servers: The Wildfly 10 CR5 release I tested against. modules: The manually compiled undertow-core versions where eb1ca6b works, but 8c8e16c does not. eclipse: An eclipse workspace of several modules, which include an ear that can be deployed, and a client which runs a test against the deployed ear. artifacts: the build ear, and client jar. To reproduce, simply deploy the ear on the Wildfly server. Start the jar, and it will keep hanging after a few minutes. When you create a thread dump of the client you will see that it hangs on socketRead0. When you deploy the "eb1ca6b" undertow-core module it will not hang. I do not seem to be able to upload the testcode. Jira keeps saying "JIRA could not attach the file as there was a missing token. Please try attaching the file again." I did try different browsers, each of them showed this. The basic idea is to create a connection pool with keep alive enabled, and then continuously call a endpoint on the Wildfly server. You will see that at some point the client side keeps hanging, and that this is undertow-core version related. I'll check in tomorrow to check if I'm able to upload by then. I just uploaded the export.zip attachment

      On the Wildfly 10 CR1 - CR4 releases we were able to keep an Apache HTTP Client connection pool to Wildfly open. On the CR5 release we see connections that are hanging (java.net.SocketInputStream.socketRead0 on the client side).

      I diagnosed this problem with building a small Java program that creates a connection pool, spawns 1 thread and tries to execute an endpoint on the Wildfly server. It keeps doing this forever, over 1 connection with keep-alive enabled. I tried to run this on CR1 - C5, like previously stated CR1 - CR4 work fine, CR5 results in a hanging connection.

      In my testing this has always happened in the first 100.000 HTTP requests. This does not happen at the same time interval, or request count. I've been playing with the Wildfly undertow parameters, but this did not seem to make a difference.

      I downgraded the bundled undertow-core dependency in the CR5 release of Wildfly 10 to 1.3.3.Final and went up from there. For 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.3.9. and 1.3.10 all seem to work fine. When I try 1.3.11 the pool freezes after a while. All threads keep hanging on (socketRead0). Upgrading to 1.3.12.Final does not resolve this issue.

      I checked out the undertow source code, and tried to find the commit that breaks the connection pool. It seems that commit "8c8e16c780a9e735ea9f90dadf1e08adb7fdb8a0" introduces this error. When I rebase the master branch and remove this commit from history, I no longer see hanging connections. When I build commit "eb1ca6b249ba89b374517940a25d93ec359c162a" manually (which is the parent of the previously stated commit), the test program won't hang. To me this means that commit "8c8e16c" introduced this bug.

      When inspecting the code, I have the feeling that the commit does what it describes to do, and I do not know how it could introduce hanging connections.

      For me it would be a shame if the final version of Wildfly 10 would be released with a undertow-core version that makes using a HTTP connection pool impossible.

              sdouglas1@redhat.com Stuart Douglas (Inactive)
              wouteraa_jira Wouter van der Aa (Inactive)
              David Lloyd, Stuart douglas (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: