Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1348

Client cert auth requests failing with "UT000124: renegotiation timed out" under load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Do
    • Icon: Major Major
    • None
    • 2.0.7.Final
    • Security, SSL
    • None
    • Hide

      Bombard Undertow with big number of parallel client cert authenticated requests.

      Show
      Bombard Undertow with big number of parallel client cert authenticated requests.

      When multiple clients all authenticating with client certs make enough requests in parallel Undertow worker task pool gets saturated and all threads in the pool are stuck at:

      "XNIO-2 task-64" #256 prio=5 os_prio=0 tid=0x00007f537c024800 nid=0x1005 in Object.wait() [0x00007f5366883000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at io.undertow.protocols.ssl.SslConduit.awaitReadable(SslConduit.java:320)

      • locked <0x00000001ce9dc0b0> (a io.undertow.protocols.ssl.SslConduit)
        at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:155)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:216)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:125)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:92)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
        at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
        at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
        at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
        at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      Then little by little SSL handshakes start falling of with:

      java.lang.IllegalStateException: UT000124: renegotiation timed out
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:222)
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:127)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:94)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
      at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
      at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
      at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
      at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      By the looks of it the place from which notifyAll() happens (SslConduit.runTasks()) has to be executed in a thread from the same saturated pool which will obviously not have any free threads to allocate.
      This causes vicious cycle where the app becomes barely responsive.

              rhn-cservice-bbaranow Bartosz Baranowski
              akikhtenko Alex Kikhtenko (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: