Uploaded image for project: 'RESTEasy'
  1. RESTEasy
  2. RESTEASY-3600

SseBroadcaster.broadcast(...) may block indefinitely due to uncompleted CompletionStage after session invalidation and undetected wrapped exception

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 6.2.15.Final, 7.0.1.Final
    • 6.2.11.Final
    • SSE
    • None
    • Hide

      Summary:

      A wrapped IllegalStateException from a session-invalidation-related failure is not detected by the SseBroadcaster, leading to an uncompleted CompletionStage entering the chain. This blocks further processing and results in undelivered events and memory leaks.

      The root cause is a combination of:

      • Session-aware I/O silently failing,
      • Lack of deep inspection of exceptions inside send(...),
      • and the broadcast chaining model depending on timely completion.

      This issue is difficult to detect at runtime and may remain unnoticed until severe resource exhaustion occurs.

      Steps:

      • A SseEventSink is associated with a session-backed HTTP connection.
      • The session is invalidated (e.g. due to timeout, logout, etc.).
      • On the next broadcast(...) call:
        • The send(...) method of the sink attempts to write to the AsyncOutputStream.
        • This fails internally due to the invalidated session and would typically throw an IllegalStateException.
        • However, the underlying ServletContainer wraps this exception into a RuntimeException, which itself is wrapped in a CompletionException.
        • The exception is not propagated as a synchronous failure, nor is it recognized by the broadcaster or the sink.
        • The send(...) method returns a CompletionStage that is never completed - it is the internal future of the WriteOperation of the AsyncOutputStream (-> DeferredOutputStream).
      • This CompletionStage is passed into the broadcaster’s internal .thenCompose(...) chain.
      • The result: the chain hangs indefinitely and all subsequent sinks never receive the broadcasted event.

      Important Technical Detail:

      In this case, the root cause (IllegalStateException) would normally lead the broadcaster to remove the sink correctly (as it does when sending to a closed sink, see SseEventOutputImpl). But:

      • The actual exception is wrapped and abstracted by the servlet container,
      • And the broadcaster does not inspect or unwrap the exception deeply,
      • Thus, the broadcaster fails to remove the faulty sink and continues including it in future broadcasts,
      • Leading to hanging sends, buffer buildup, and eventually OutOfMemoryError.

      Workaround (Limited):

      We detect failures using the SseBroadcaster.onError(...) mechanism and close the affected sink manually. On the next event, send(...) then throws a synchronous exception, allowing the broadcaster to cleanly remove the sink.

      However, this only works if:

      • The send(...) method actually throws or the stage completes exceptionally.

      In the scenario described above, neither occurs — instead, the stage remains pending forever.

      Show
      Summary: A wrapped IllegalStateException from a session-invalidation-related failure is not detected by the SseBroadcaster , leading to an uncompleted CompletionStage entering the chain. This blocks further processing and results in undelivered events and memory leaks. The root cause is a combination of: Session-aware I/O silently failing, Lack of deep inspection of exceptions inside send(...) , and the broadcast chaining model depending on timely completion. This issue is difficult to detect at runtime and may remain unnoticed until severe resource exhaustion occurs. Steps: A SseEventSink is associated with a session-backed HTTP connection. The session is invalidated (e.g. due to timeout, logout, etc.). On the next broadcast(...) call: The send(...) method of the sink attempts to write to the AsyncOutputStream . This fails internally due to the invalidated session and would typically throw an IllegalStateException . However, the underlying ServletContainer wraps this exception into a RuntimeException , which itself is wrapped in a CompletionException. The exception is not propagated as a synchronous failure , nor is it recognized by the broadcaster or the sink. The send(...) method returns a CompletionStage that is never completed - it is the internal future of the WriteOperation of the AsyncOutputStream (-> DeferredOutputStream). This CompletionStage is passed into the broadcaster’s internal .thenCompose(...) chain. The result: the chain hangs indefinitely and all subsequent sinks never receive the broadcasted event . Important Technical Detail: In this case, the root cause ( IllegalStateException ) would normally lead the broadcaster to remove the sink correctly (as it does when sending to a closed sink, see SseEventOutputImpl ). But: The actual exception is wrapped and abstracted by the servlet container , And the broadcaster does not inspect or unwrap the exception deeply, Thus, the broadcaster fails to remove the faulty sink and continues including it in future broadcasts, Leading to hanging sends , buffer buildup , and eventually OutOfMemoryError . Workaround (Limited): We detect failures using the SseBroadcaster.onError(...) mechanism and close the affected sink manually. On the next event, send(...) then throws a synchronous exception, allowing the broadcaster to cleanly remove the sink. However, this only works if: The send(...) method actually throws or the stage completes exceptionally. In the scenario described above, neither occurs — instead, the stage remains pending forever.

      When using RESTEasy’s SseBroadcaster, we’ve observed a critical issue that causes broadcast(...) to block indefinitely and skip downstream sinks if a registered SseEventSink returns a CompletionStage that is never completed.

      This behavior occurs after session invalidation, due to how AsyncOutputStream and the surrounding container handle write failures.

      Real-world Impact:

      • Partial delivery of events (some clients receive events, others don’t).
      • Silent buffering for a dead sink.
      • Increasing memory usage.
      • Ultimately: out of memory error of the application server, affecting the whole server cluster.

      Environment:

      • Application server: RedHat JBoss EAP 8.0.7
      • Servlet container: Undertow
      • Java version: Azul JDK 17.0.15

              jperkins-rhn James Perkins
              jung-soptim Volker Godas
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: