Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1074

Deadlock in Http2Channel

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.0.0.Beta1, 1.4.15.Final
    • 1.4.11.Final
    • None
    • None

      We observed the following deadlock related to handling http2 requests:

      Found one Java-level deadlock:
      =============================
      "XNIO-1 task-198":
        waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel),
        which is held by "XNIO-1 I/O-1"
      "XNIO-1 I/O-1":
        waiting to lock monitor 0x00007ff390085128 (object 0x00000006f2286a10, a java.lang.Object),
        which is held by "XNIO-1 task-105"
      "XNIO-1 task-105":
        waiting to lock monitor 0x00007ff3c41c0498 (object 0x00000006f0aefa68, a io.undertow.protocols.http2.Http2Channel),
        which is held by "XNIO-1 I/O-1"
      
      Java stack information for the threads listed above:
      ===================================================
      "XNIO-1 task-198":
      	at io.undertow.protocols.http2.Http2Channel.updateReceiveFlowControlWindow(Http2Channel.java:789)
      	- waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
      	at io.undertow.protocols.http2.Http2StreamSourceChannel.updateFlowControlWindow(Http2StreamSourceChannel.java:182)
      	at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:142)
      	at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43)
      	at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
      	at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
      	at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287)
      	at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47)
      	at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      "XNIO-1 I/O-1":
      	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.markStreamBroken(AbstractFramedStreamSourceChannel.java:658)
      	- waiting to lock <0x00000006f2286a10> (a java.lang.Object)
      	at io.undertow.server.protocol.framed.AbstractFramedChannel.markReadsBroken(AbstractFramedChannel.java:822)
      	at io.undertow.server.protocol.framed.AbstractFramedChannel.receive(AbstractFramedChannel.java:474)
      	- locked <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
      	at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:106)
      	at io.undertow.server.protocol.http2.Http2ReceiveListener.handleEvent(Http2ReceiveListener.java:57)
      	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
      	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:931)
      	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:912)
      	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
      	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
      	at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1129)
      	at io.undertow.protocols.ssl.SslConduit$1.run(SslConduit.java:168)
      	at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:580)
      	at org.xnio.nio.WorkerThread.run(WorkerThread.java:464)
      "XNIO-1 task-105":
      	at io.undertow.server.protocol.framed.AbstractFramedChannel.notifyClosed(AbstractFramedChannel.java:903)
      	- waiting to lock <0x00000006f0aefa68> (a io.undertow.protocols.http2.Http2Channel)
      	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.exitRead(AbstractFramedStreamSourceChannel.java:584)
      	- locked <0x00000006f2286a10> (a java.lang.Object)
      	at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.read(AbstractFramedStreamSourceChannel.java:531)
      	at io.undertow.protocols.http2.Http2StreamSourceChannel.read(Http2StreamSourceChannel.java:141)
      	at org.xnio.conduits.StreamSourceChannelWrappingConduit.read(StreamSourceChannelWrappingConduit.java:43)
      	at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
      	at io.undertow.channels.DetachableStreamSourceChannel.read(DetachableStreamSourceChannel.java:209)
      	at io.undertow.server.HttpServerExchange$ReadDispatchChannel.read(HttpServerExchange.java:2287)
      	at io.undertow.io.AsyncReceiverImpl.receivePartialBytes(AsyncReceiverImpl.java:579)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.readMessage(MessageReadHandler.java:90)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.MessageReadHandler.handleRequest(MessageReadHandler.java:61)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.lambda$handleRequest$0(TopicHandler.java:51)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler$$Lambda$124/173628928.accept(Unknown Source)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.onRequestValid(TopicHandler.java:79)
      	at pl.allegro.tech.hermes.frontend.publishing.handlers.TopicHandler.handleRequest(TopicHandler.java:47)
      	at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:52)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:211)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:809)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Do you have an idea what could have caused it?

      Not sure if it's related, but at the same time we observed increasing number of leaked file descriptors ("can't identify protocol" descriptors and CLOSE_WAIT sockets in lsof output), ~40k at the time this thread dump was made. Both the deadlock and leaked file descriptors resolved after application restart.

              sdouglas1@redhat.com Stuart Douglas (Inactive)
              pbetkier Piotr Betkier (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: