Uploaded image for project: 'Undertow'
  1. Undertow
  2. UNDERTOW-1348

Client cert auth requests failing with "UT000124: renegotiation timed out" under load

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • 2.0.7.Final
    • Security, SSL
    • None
    • Hide

      Bombard Undertow with big number of parallel client cert authenticated requests.

      Show
      Bombard Undertow with big number of parallel client cert authenticated requests.

    Description

      When multiple clients all authenticating with client certs make enough requests in parallel Undertow worker task pool gets saturated and all threads in the pool are stuck at:

      "XNIO-2 task-64" #256 prio=5 os_prio=0 tid=0x00007f537c024800 nid=0x1005 in Object.wait() [0x00007f5366883000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at io.undertow.protocols.ssl.SslConduit.awaitReadable(SslConduit.java:320)

      • locked <0x00000001ce9dc0b0> (a io.undertow.protocols.ssl.SslConduit)
        at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:155)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:216)
        at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:125)
        at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:92)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
        at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
        at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
        at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
        at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
        at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:332)
        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      Then little by little SSL handshakes start falling of with:

      java.lang.IllegalStateException: UT000124: renegotiation timed out
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiateNoRequest(ConnectionSSLSessionInfo.java:222)
      at io.undertow.server.ConnectionSSLSessionInfo.renegotiate(ConnectionSSLSessionInfo.java:130)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.getPeerCertificates(ClientCertAuthenticationMechanism.java:127)
      at io.undertow.security.impl.ClientCertAuthenticationMechanism.authenticate(ClientCertAuthenticationMechanism.java:94)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:245)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.transition(SecurityContextImpl.java:268)
      at io.undertow.security.impl.SecurityContextImpl$AuthAttempter.access$100(SecurityContextImpl.java:231)
      at io.undertow.security.impl.SecurityContextImpl.attemptAuthentication(SecurityContextImpl.java:125)
      at io.undertow.security.impl.SecurityContextImpl.authTransition(SecurityContextImpl.java:99)
      at io.undertow.security.impl.SecurityContextImpl.authenticate(SecurityContextImpl.java:92)
      at io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(AuthenticationCallHandler.java:50)
      at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
      at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      By the looks of it the place from which notifyAll() happens (SslConduit.runTasks()) has to be executed in a thread from the same saturated pool which will obviously not have any free threads to allocate.
      This causes vicious cycle where the app becomes barely responsive.

      Attachments

        Activity

          People

            Unassigned Unassigned
            akikhtenko Alex Kikhtenko (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: