Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-27971

[GSS](8.0.z) UNDERTOW-2436 - Race condition for HttpServerExchange state allows missed FLAG_REQUEST_TERMINATED flag with async requests and subsequent connection stall

XMLWordPrintable

    • False
    • None
    • False
    • Workaround Exists
    • Hide

      Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request:

      /subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=connection-close-header:add
      Show
      Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request: /subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close) /subsystem=undertow/server= default -server/host= default -host/filter-ref=connection-close-header:add

      The HttpServerExchange state has race conditions that can allow for issues with async requests. For instance from some byteman tracing, note how the setInCall and terminateRequest method can be executed in different threads concurrently with async requests:

      2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) BTM setInCall false HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange.setInCall(HttpServerExchange.java:-1)
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.Connectors.executeRootHandler(Connectors.java:396)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:855)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) java.lang.Thread.run(Thread.java:750)

2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) BTM terminateRequest HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange.terminateRequest(HttpServerExchange.java:-1)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.Connectors.terminateRequest(Connectors.java:180)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:179)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:172)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.invokeFinishListener(FixedLengthStreamSourceConduit.java:409)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:253)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:214)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2446)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.channels.Channels.readBlocking(Channels.java:344)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:201)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:162)

      With the right timing through those methods, this can result in setInCall's FLAG_IN_CALL removal completely clobbering terminateRequest's attempt to set the FLAG_REQUEST_TERMINATED flag. After such a request completes, this connection is then left in a bad state with that exchange still lingering in an incomplete state (state value=19656, or 100110011001000, with 14 FLAG_PERSISTENT yes, 12 FLAG_REQUEST_TERMINATED no, 11 FLAG_RESPONSE_TERMINATED yes, 10 FLAG_RESPONSE_SENT yes) so the connection will not begin to process any next request after this. Any client/proxy then times out when reusing that connection and the connection is eventually left as a CLOSE_WAIT on JBoss.

              flaviarnn Flavia Rainone
              rhn-support-aogburn Aaron Ogburn
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: