-
Bug
-
Resolution: Done
-
Major
-
1.4.11.Final
-
None
-
None
We observed the following deadlock related to handling http2 requests:
Found one Java-level deadlock: ============================= "XNIO-1 task-198": waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel), which is held by "XNIO-1 I/O-1" "XNIO-1 I/O-1": waiting to lock monitor 0x00007ff390085128 (object 0x00000006f2286a10, a java.lang.Object), which is held by "XNIO-1 task-105" "XNIO-1 task-105": waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel), which is held by "XNIO-1 I/O-1" Java stack information for the threads listed above: =================================================== "XNIO-1 task-198": at io.undertow.protocols.http2.Http2Channel.updateReceiveFlowControlWindow(Http2Channel.java:789) - waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel) at io.undertow.protocols.http2.Http2StreamSourceChannel.updateFlowControlWindow(Http2StreamSourceChannel.java:182) at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:142) at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43) at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209) at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287) at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579) at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90) at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47) at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "XNIO-1 I/O-1": at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.markStreamBroken(AbstractFramedStreamSourceChannel.java:658) - waiting to lock <0x00000006f2286a10> (a java.lang.Object) at io.undertow.server.protocol.framed.AbstractFramedChannel.markReadsBroken(AbstractFramedChannel.java:822) at io.undertow.server.protocol.framed.AbstractFramedChannel.receive(AbstractFramedChannel.java:474) - locked <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel) at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:106) at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:57) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:931) at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:912) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1129) at io.undertow.protocols.ssl.SslConduit$1.run(SslConduit.java:168) at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:580) at org.xnio.nio.WorkerThread.run(WorkerThread.java:464) "XNIO-1 task-105": at io.undertow.server.protocol.framed.AbstractFramedChannel.notifyClosed(AbstractFramedChannel.java:903) - waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel) at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.exitRead(AbstractFramedStreamSourceChannel.java:584) - locked <0x00000006f2286a10> (a java.lang.Object) at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.read(AbstractFramedStreamSourceChannel.java:531) at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:141) at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43) at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209) at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287) at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579) at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90) at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79) at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47) at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Do you have an idea what could have caused it?
Not sure if it's related, but at the same time we observed increasing number of leaked file descriptors ("can't identify protocol" descriptors and CLOSE_WAIT sockets in lsof output), ~40k at the time this thread dump was made. Both the deadlock and leaked file descriptors resolved after application restart.
- is related to
-
JBEAP-15330 [GSS](7.1.z) UNDERTOW-1231/UNDERTOW-1179 - Deadlock AbstractFramedChannel when notifyClosed and markStreamBroken are called related to Http2Channel
- Closed
-
UNDERTOW-1231 Deadlock AbstractFramedChannel when notifyClosed and markStreamBroken are called related to Http2Channel
- Closed