Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1348

Client cert auth requests failing with "UT000124: renegotiation timed out" under load

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.0.7.Final
    • Fix Version/s: None
    • Component/s: Security, SSL
    • Labels:
      None
    • Environment:

      jdk1.8.0u172

    • Steps to Reproduce:
      Hide

      Bombard Undertow with big number of parallel client cert authenticated requests.

      Show
      Bombard Undertow with big number of parallel client cert authenticated requests.

      Description

      When multiple clients all authenticating with client certs make enough requests in parallel Undertow worker task pool gets saturated and all threads in the pool are stuck at:

      "XNIO-2 task-64" #256 prio=5 os_prio=0 tid=0x00007f537c024800 nid=0x1005 in Object.wait() [0x00007f5366883000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at io.undertow.protocols.ssl.SslConduit.awaitReadable(SslConduit.java:320)

      • locked <0x00000001ce9dc0b0> (a io.undertow.protocols.ssl.SslConduit)
        at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:155)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:216)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:125)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:92)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
        at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
        at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
        at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
        at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      Then little by little SSL handshakes start falling of with:

      java.lang.IllegalStateException: UT000124: renegotiation timed out
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:222)
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:127)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:94)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
      at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
      at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
      at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
      at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      By the looks of it the place from which notifyAll() happens (SslConduit.runTasks()) has to be executed in a thread from the same saturated pool which will obviously not have any free threads to allocate.
      This causes vicious cycle where the app becomes barely responsive.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                swd847 Stuart Douglas
                Reporter:
                akikhtenko Alex Kikhtenko
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: