Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1074

Deadlock in Http2Channel

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 2.0.0.Beta1, 1.4.15.Final
    • 1.4.11.Final
    • None
    • None

    Description

      We observed the following deadlock related to handling http2 requests:

      Found one Java-level deadlock:
=============================
"XNIO-1 task-198":
  waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel),
  which is held by "XNIO-1 I/O-1"
"XNIO-1 I/O-1":
  waiting to lock monitor 0x00007ff390085128 (object 0x00000006f2286a10, a java.lang.Object),
  which is held by "XNIO-1 task-105"
"XNIO-1 task-105":
  waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel),
  which is held by "XNIO-1 I/O-1"

Java stack information for the threads listed above:
===================================================
"XNIO-1 task-198":
	at io.undertow.protocols.http2.Http2Channel.updateReceiveFlowControlWindow(Http2Channel.java:789)
	- waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
	at io.undertow.protocols.http2.Http2StreamSourceChannel.updateFlowControlWindow(Http2StreamSourceChannel.java:182)
	at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:142)
	at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43)
	at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
	at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
	at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287)
	at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47)
	at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
"XNIO-1 I/O-1":
	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.markStreamBroken(AbstractFramedStreamSourceChannel.java:658)
	- waiting to lock <0x00000006f2286a10> (a java.lang.Object)
	at io.undertow.server.protocol.framed.AbstractFramedChannel.markReadsBroken(AbstractFramedChannel.java:822)
	at io.undertow.server.protocol.framed.AbstractFramedChannel.receive(AbstractFramedChannel.java:474)
	- locked <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
	at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:106)
	at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:57)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:931)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:912)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1129)
	at io.undertow.protocols.ssl.SslConduit$1.run(SslConduit.java:168)
	at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:580)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:464)
"XNIO-1 task-105":
	at io.undertow.server.protocol.framed.AbstractFramedChannel.notifyClosed(AbstractFramedChannel.java:903)
	- waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.exitRead(AbstractFramedStreamSourceChannel.java:584)
	- locked <0x00000006f2286a10> (a java.lang.Object)
	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.read(AbstractFramedStreamSourceChannel.java:531)
	at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:141)
	at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43)
	at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
	at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
	at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287)
	at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79)
	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47)
	at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

      Do you have an idea what could have caused it?

      Not sure if it's related, but at the same time we observed increasing number of leaked file descriptors ("can't identify protocol" descriptors and CLOSE_WAIT sockets in lsof output), ~40k at the time this thread dump was made. Both the deadlock and leaked file descriptors resolved after application restart.

      Attachments

        Issue Links

          is related to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-15330 [GSS](7.1.z) UNDERTOW-1231/UNDERTOW-1179 - Deadlock AbstractFramedChannel when notifyClosed and markStreamBroken are called related to Http2Channel

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. UNDERTOW-1231 Deadlock AbstractFramedChannel when notifyClosed and markStreamBroken are called related to Http2Channel

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              sdouglas1@redhat.com Stuart Douglas
              pbetkier Piotr Betkier (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: