Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-14853

Concurrent invalidation requests can cause memory leak in ConcurrentManager, and prevent creation of a new session

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 24.0.0.Final
    • 23.0.2.Final
    • Clustering
    • None
    • Hide

      Either of the following:

      • Use transactional cache instead of non-tx cache (revert to enable <locking isolation="REPEATABLE_READ"/> and <transaction mode="BATCH"/>)
      • Disable session replication (remove "<distributable/>" from web.xml)
      Show
      Either of the following: Use transactional cache instead of non-tx cache (revert to enable <locking isolation="REPEATABLE_READ"/> and <transaction mode="BATCH"/>) Disable session replication (remove "<distributable/>" from web.xml)
    • Undefined

    Description

      This issue happens after a client initiates two requests that invoke "session.invalidate()" concurrently; for example, double-click a logout button accidentally.
      (As an error response can be returned in this situation, the client can still hold the invalidated session id in the cookie.)

      Once this happens, every time client sends a request having a request cookie with the invalidated sessionId, the server does not respond with a new session but it returns the error with "IllegalStateException: WFLYCLWEBUT0001: Session <sessionId> is invalid". The client needs to clear the browser cookie to login to the application again. It's expected that the server should return a new session instead of throwing the exception in this situation.


      I could reproduce the same issue with the simple servlet application in my local environment.
      You can reproduce the issue with the following steps with the attached example application (example.war).

      1. Send a request that generates a new cookie
      (query parameter "n" specifies number of session attributes.
      the example app generates 100 session attributes by default.
      so specify n=1 for this test because a number of session attributes is not related here.)

      curl -0 -v -b /tmp/curl_cookie.txt -c /tmp/curl_cookie.txt http://${SERVER_NAME}:8080/example/new?n=1
      

      2. Send a request that invalidates the session twice.
      the 2nd one will fail with "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid".

      curl -0 -v -b /tmp/curl_cookie.txt -c /tmp/curl_cookie.txt http://${SERVER_NAME}:8080/example/invalidate & curl -0 -v -b /tmp/curl_cookie.txt -c /tmp/curl_cookie.txt http://${SERVER_NAME}:8080/example/invalidate
      

      3. Send a request with the invalidated session cookie.

      curl -0 -v -b /tmp/curl_cookie.txt -c /tmp/curl_cookie.txt http://${SERVER_NAME}:8080/example/new?n=1
      

      we expect the server should return a new session cookie, but this request will fail with the following "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid" after the above 2nd one failed. And, when you repeat this request, "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid" happens on the server every time.

      example output of "IllegalStateException: WFLYCLWEBUT0001: Session <sessionId> is invalid"
      08:36:45,499 ERROR [io.undertow.request] (default task-2) UT005023: Exception handling request to /example/new: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session 0nLsDAWIQ4z6Ah6p_hQn2HeDDMmO65EXjGMeQngk is invalid
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:293)
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:285)
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:161)
      	at io.undertow.servlet.util.SavedRequest.tryRestoreRequest(SavedRequest.java:155)
      	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:74)
      	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:64)
      	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
      	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
      	at java.lang.Thread.run(Thread.java:748)
      
      08:37:08,752 ERROR [io.undertow.request] (default task-2) UT005023: Exception handling request to /example/new: java.lang.IllegalStateException: WFLYCLWEBUT0001: Session 0nLsDAWIQ4z6Ah6p_hQn2HeDDMmO65EXjGMeQngk is invalid
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:293)
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.validate(DistributableSession.java:285)
      	at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:161)
      	at io.undertow.servlet.util.SavedRequest.tryRestoreRequest(SavedRequest.java:155)
      	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:74)
      	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
      	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      	at io.undertow.server.handlers.MetricsHandler.handleRequest(MetricsHandler.java:64)
      	at io.undertow.servlet.core.MetricsChainHandler.handleRequest(MetricsChainHandler.java:59)
      	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:78)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:133)
      	at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:130)
      	at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
      	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:249)
      	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:78)
      	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:99)
      	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:370)
      	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
      	at java.lang.Thread.run(Thread.java:748)
      

      You can repeat the above with the following script:

      test.sh
      #!/bin/bash
      
      COOKIE_FILE="/tmp/curl_test_cookie.txt"
      # SERVER_NAME="localhost"
      SERVER_NAME="node1"
      
      count=0
      
      while true
      do
        # clear COOKIE_FILE every iteration
        if  [ -f ${COOKIE_FILE} ]; then
          rm -f ${COOKIE_FILE}
        fi
      
        # send a request that generates a new cookie
        # (query parameter "n" specifies number of session attributes.)
        # the example app generates 100 session attributes by default.
        # so specify n=1 for this test because number of session attributes is not related here.
        curl -0 -v -b ${COOKIE_FILE} -c ${COOKIE_FILE} http://${SERVER_NAME}:8080/example/new?n=1 >> /tmp/sessionid_result.txt
      
        # send a request that invalidates session twice.
        # the 2nd one will fail with "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid".
        curl -0 -v -b ${COOKIE_FILE} -c ${COOKIE_FILE} http://${SERVER_NAME}:8080/example/invalidate & curl -0 -v -b ${COOKIE_FILE} -c ${COOKIE_FILE} http://${SERVER_NAME}:8080/example/invalidate
      
        sleep 1
      
        # send a requst with invalidated session cookie.
        # we expect the server should return a new session cookie,
        # but this request will fail with "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid" after the above 2nd one failed.
        curl -0 -v -b ${COOKIE_FILE} -c ${COOKIE_FILE} http://${SERVER_NAME}:8080/example/new?n=1
      
        echo "#####"
        echo "##### loop count = $((++count))"
        echo "#####"
      
        sleep 2
      done
      

      When I captured a heapdump after reproducing the issue, it appears that a session object (and session attributes) is invalidated correctly but a session-related MetaData (like org.wildfly.clustering.web.cache.session.SimpleSessionCreationMetaData) still remains.
      The remaining SimpleSessionCreationMetaData instances hold "valid = 0 (false)". So, I think this is why "IllegalStateException: WFLYCLWEBUT0001: Session xxx is invalid" is returned for the session repeatedly. I do not know why these MetaData still remain and not cleaned up.
      I guess this issue can be treated as a leak of session-related MetaData. Of course, the size of MetaData is relatively small, so I guess this MetaData itself does not have a huge impact.

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: