Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-712

Randomly ServerSentEventConnection$SseWriteListener.handleEvent goes into an infinite loop

    Details

    • Steps to Reproduce:
      Hide

      it happens at random, usually after few hours of normal run.

      Show
      it happens at random, usually after few hours of normal run.

      Description

      We have noticed on our production servers an abnormal CPU usage which happens ramdomly after few minutes to hours of normal run. The only way to recover from this situation is to restart the server. Even after disconnecting all the clients connected to the server, the CPU is still high.
      I took a thread dump before restarting the server. It seems that a thread is running in an infinite loop:

      "proxy-xnio I/O-3" #31 prio=5 os_prio=0 tid=0x00007ff8b93c0800 nid=0x67b4 runnable [0x00007ff8941ad000]
         java.lang.Thread.State: RUNNABLE
          at sun.nio.ch.NativeThread.current(Native Method)
          at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:468)
          - locked <0x00000000840382e0> (a java.lang.Object)
          - locked <0x00000000840382d0> (a java.lang.Object)
          at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:152)
          at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:599)
          at org.xnio.conduits.AbstractStreamSinkConduit.write(AbstractStreamSinkConduit.java:51)
          at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
          at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)
          at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2004)
          at io.undertow.server.handlers.sse.ServerSentEventConnection$SseWriteListener.handleEvent(ServerSentEventConnection.java:510)
          - locked <0x0000000084037eb0> (a io.undertow.server.handlers.sse.ServerSentEventConnection)
          at io.undertow.server.handlers.sse.ServerSentEventConnection$SseWriteListener.handleEvent(ServerSentEventConnection.java:475)
          at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
          at io.undertow.channels.DetachableStreamSinkChannel$SetterDelegatingListener.handleEvent(DetachableStreamSinkChannel.java:285)
          at io.undertow.channels.DetachableStreamSinkChannel$SetterDelegatingListener.handleEvent(DetachableStreamSinkChannel.java:272)
          at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
          at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65)
          at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:93)
          at org.xnio.nio.WorkerThread.run(WorkerThread.java:559)
      

      I tried to reproduce it on my local test bed with few clients connected. It took hours but I finally managed to do it.
      After debugging it appears that :

      In DetachableStreamSinkChannel.java line 240

      @Override
          public int write(final ByteBuffer src) throws IOException {
              if (isFinished()) {
                  throw UndertowMessages.MESSAGES.channelIsClosed();
              }
              return delegate.write(src);
          }
      

      delegate.write(src) returns 0 which creates an infinite loop in ServerSentEventConnection$SseWriteListener.handleEvent()

      The input src byteBuffer when this issue occurs is in the following state (given by intellij):

      delegate.write(src)    src: java.nio.DirectByteBuffer[pos=0 lim=0 cap=16384]
      

      After spending a lot of time on google to find out a solution to this issue, I found this undertow issue (https://issues.jboss.org/browse/UNDERTOW-282) which seems very similar to mine but occurring on AbstractStreamSourceConduit.read() instead of AbstractStreamSinkConduit.write() (which is normal since we are using an SSE connection to send data to clients)

      I tried the workaround mentioned in this issue which consists to detect when the "write" returns 0 consecutively and close the underlying connection by registering a custom ResponseWrapper in the HttpExchange of my handler.

      It works great as a workaround.
      I think that the fix applied for the read() on ISSUE UNDERTOW-282 should be also applied to the write()

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  swd847 Stuart Douglas
                  Reporter:
                  ericp2 Eric P
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: