Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-2436

Race condition for HttpServerExchange state allows missed FLAG_REQUEST_TERMINATED flag with async requests and subsequent connection stall

XMLWordPrintable

    • Workaround Exists
    • Hide

      Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request:

      /subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=connection-close-header:add
      Show
      Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request: /subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close) /subsystem=undertow/server= default -server/host= default -host/filter-ref=connection-close-header:add

      The HttpServerExchange state has race conditions that can allow for issues with async requests. For instance from some byteman tracing, note how the setInCall and terminateRequest method can be executed in different threads concurrently with async requests:

      2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) BTM setInCall false HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange.setInCall(HttpServerExchange.java:-1)
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.Connectors.executeRootHandler(Connectors.java:396)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:855)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) java.lang.Thread.run(Thread.java:750)

2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) BTM terminateRequest HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange.terminateRequest(HttpServerExchange.java:-1)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.Connectors.terminateRequest(Connectors.java:180)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:179)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:172)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.invokeFinishListener(FixedLengthStreamSourceConduit.java:409)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:253)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:214)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2446)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.channels.Channels.readBlocking(Channels.java:344)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:201)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:162)

      With the right timing through those methods this can result in setInCall's FLAG_IN_CALL removal completely clobbering terminateRequest's attempt to set the FLAG_REQUEST_TERMINATED flag. After that request completes, this connection is then left in a bad state with that exchange still lingering in an incomplete state (state value =19656, 14 FLAG_PERSISTENT, 12 FLAG_REQUEST_TERMINATED no, 11 FLAG_RESPONSE_TERMINATED yes, 10 FLAG_RESPONSE_SENT yes) so the connection will not begin to process any next request. Any client/proxy then times out when reusing that connection and it's eventually left as a CLOSE_WAIT on JBoss.

            flaviarnn Flavia Rainone
            rhn-support-aogburn Aaron Ogburn
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: