Loading...

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: 2.3.18.Final, 2.2.37.Final
Affects Version/s: 2.3.19.Final, 2.2.38.Final
Component/s: Core
Labels:
None

Workaround:

Workaround Exists
Workaround Description:
Hide

Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request:

/subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close) /subsystem=undertow/server=default-server/host=default-host/filter-ref=connection-close-header:add
Show
Fully disabling keep-alive connection reuse avoids the possibility of this impacting a request: /subsystem=undertow/configuration=filter/response-header=connection-close-header:add(header-name=Connection,header-value=close) /subsystem=undertow/server= default -server/host= default -host/filter-ref=connection-close-header:add
Git Pull Request:
https://github.com/undertow-io/undertow/pull/1670, https://github.com/undertow-io/undertow/pull/1661, https://github.com/undertow-io/undertow/pull/1671, https://github.com/undertow-io/undertow/pull/1687, https://github.com/undertow-io/undertow/pull/1689

The HttpServerExchange state has race conditions that can allow for issues with async requests. For instance from some byteman tracing, note how the setInCall and terminateRequest method can be executed in different threads concurrently with async requests:

2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) BTM setInCall false HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange.setInCall(HttpServerExchange.java:-1)
2024-09-05 12:04:01,390 INFO  [stdout] (default task-2) io.undertow.server.Connectors.executeRootHandler(Connectors.java:396)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:855)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
2024-09-05 12:04:01,391 INFO  [stdout] (default task-2) java.lang.Thread.run(Thread.java:750)

2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) BTM terminateRequest HttpServerExchange{ POST /app} 180424
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange.terminateRequest(HttpServerExchange.java:-1)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.Connectors.terminateRequest(Connectors.java:180)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:179)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.protocol.http.HttpTransferEncoding$1.handleEvent(HttpTransferEncoding.java:172)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.invokeFinishListener(FixedLengthStreamSourceConduit.java:409)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.conduits.FixedLengthStreamSourceConduit.read(FixedLengthStreamSourceConduit.java:253)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:214)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2446)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) org.xnio.channels.Channels.readBlocking(Channels.java:344)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:201)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
2024-09-05 12:04:01,391 INFO  [stdout] (pool-22-thread-2) io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:162)

With the right timing through those methods this can result in setInCall's FLAG_IN_CALL removal completely clobbering terminateRequest's attempt to set the FLAG_REQUEST_TERMINATED flag. After that request completes, this connection is then left in a bad state with that exchange still lingering in an incomplete state (state value =19656, 14 FLAG_PERSISTENT, 12 FLAG_REQUEST_TERMINATED no, 11 FLAG_RESPONSE_TERMINATED yes, 10 FLAG_RESPONSE_SENT yes) so the connection will not begin to process any next request. Any client/proxy then times out when reusing that connection and it's eventually left as a CLOSE_WAIT on JBoss.

is cloned by

JBEAP-27970 [GSS](7.4.z) UNDERTOW-2436 - Race condition for HttpServerExchange state allows missed FLAG_REQUEST_TERMINATED flag with async requests and subsequent connection stall

Verified

JBEAP-27971 [GSS](8.0.z) UNDERTOW-2436 - Race condition for HttpServerExchange state allows missed FLAG_REQUEST_TERMINATED flag with async requests and subsequent connection stall

Verified

is incorporated by

WFCORE-7030 Upgrade Undertow from 2.3.17 to 2.3.18

Resolved

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates