Uploaded image for project: 'WildFly OpenSSL'
  1. WildFly OpenSSL
  2. WFSSL-105

TLS 1.3 handshake can fall into infinite loop on server

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 2.2.5.Final
    • None
    • None
    • False
    • None
    • False

      After enabling TLS 1.3 support I found that connections from some clients will result in the server thread never exiting OpenSSL.runHandshake(), resulting in the thread utilizing 100% CPU until the application is terminated.

      I found that when this happens, calling getStatus() on the SSLEngineResult object returned by OpenSSLEngine.unwrap() returns Status.OK without any data being read from the passed in buffer. This falls through the loop termination conditions. However, getHandshakeStatus() will return HandshakeStatus.NEED_WRAP, indicating that a write is needed first.

      Adding a check for HandshakeStatus.NEED_WRAP as a condition for breaking out of the read loop appears to fix the issue and allow TLS 1.3 handshakes to proceed normally.

              Unassigned Unassigned
              howie.richmond Howie Richmond (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: