Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-29780

(8.1.z) [CLUSTERING] Intermittent "WFLYCLWEBUT0003: Session manager was stopped" when server is shut down

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 8.1.0.Beta
    • Clustering
    • False
    • Hide

      None

      Show
      None
    • False
    • Known Issue

      Scenario: 4 node cluster with HAPROXY load balancer (config in HAPROXY-logs-and-config.zip ) and replicated cache:

      embed-server --server-config=standalone-ha.xml
if (outcome != success) of /subsystem=jgroups:read-attribute(name=default-stack)
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
else
/subsystem=jgroups:write-attribute(name=default-stack,value=tcp)
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
end-if
batch
/subsystem=infinispan/cache-container=web/distributed-cache=dist:remove
/subsystem=infinispan/cache-container=ejb/distributed-cache=dist:remove
# web cache
/subsystem=infinispan/cache-container=web/replicated-cache=dist:add()
/subsystem=infinispan/cache-container=web/replicated-cache=dist/component=locking:add(isolation=REPEATABLE_READ)
/subsystem=infinispan/cache-container=web/replicated-cache=dist/component=transaction:add(mode=BATCH)
/subsystem=infinispan/cache-container=web/replicated-cache=dist/store=file:add(purge=true, passivation=true)
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=dist)
# ejb cache
/subsystem=infinispan/cache-container=ejb/replicated-cache=dist:add()
/subsystem=infinispan/cache-container=ejb/replicated-cache=dist/component=locking:add(isolation=REPEATABLE_READ)
/subsystem=infinispan/cache-container=ejb/replicated-cache=dist/component=transaction:add(mode=BATCH)
/subsystem=infinispan/cache-container=ejb/replicated-cache=dist/store=file:add(purge=true, passivation=true)
/subsystem=infinispan/cache-container=ejb:write-attribute(name=default-cache, value=dist)
run-batch

      When Node 3 is shut down to trigger fail over, we see the following error (complete logs in eap-8.1.0.Beta-CR9.zip ):

      2025-03-25 23:29:08,535 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
2025-03-25 23:29:08,554 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
2025-03-25 23:29:08,558 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 86) [Context=org.infinispan.CONFIG] ISPN100008: Updating cache members list [wildfly4, wildfly1, wildfly2], topology id 36
2025-03-25 23:29:08,559 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 86) [Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [wildfly4, wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 37
2025-03-25 23:29:08,561 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 87) [Context=org.infinispan.CONFIG] ISPN100008: Updating cache members list [wildfly4, wildfly1, wildfly2], topology id 36
2025-03-25 23:29:08,561 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 87) [Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [wildfly4, wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 37
2025-03-25 23:29:08,565 INFO  [org.infinispan.CONTAINER] (ServerService Thread Pool -- 86) ISPN000390: Persisted state, version=15.0.14.Final-redhat-00002 timestamp=2025-03-25T23:29:08.564688801Z
2025-03-25 23:29:08,568 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 86) Stopped hibernate cache container
2025-03-25 23:29:08,570 INFO  [org.infinispan.CONTAINER] (ServerService Thread Pool -- 87) ISPN000390: Persisted state, version=15.0.14.Final-redhat-00002 timestamp=2025-03-25T23:29:08.570215746Z
2025-03-25 23:29:08,570 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 88) WFLYUT0022: Unregistered web context: '/clusterbench-passivating' from server 'default-server'
2025-03-25 23:29:08,571 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) Stopped server cache container
2025-03-25 23:29:08,572 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 86) WFLYUT0022: Unregistered web context: '/clusterbench' from server 'default-server'
2025-03-25 23:29:08,583 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 92) [Context=clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default] ISPN100008: Updating cache members list [wildfly4, wildfly1, wildfly2], topology id 34
2025-03-25 23:29:08,584 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 92) [Context=clusterbench-ee10.ear/clusterbench-ee10-ejb.jar/default] ISPN100002: Starting rebalance with members [wildfly4, wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 35
2025-03-25 23:29:08,587 ERROR [io.undertow.servlet.request] (default task-69) UT015005: Error invoking method requestInitialized on listener class org.jboss.weld.module.web.servlet.WeldInitialListener: java.lang.IllegalStateException: WFLYCLWEBUT0003: Session manager was stopped
	at org.wildfly.clustering.web.undertow@8.1.0.Beta-redhat-00006//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSessionCloseTask(DistributableSessionManager.java:103)
	at org.wildfly.clustering.web.undertow@8.1.0.Beta-redhat-00006//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:201)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:884)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:454)
	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:266)
	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:151)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:263)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at org.wildfly.extension.undertow@8.1.0.Beta-redhat-00006//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
	at org.wildfly.extension.undertow@8.1.0.Beta-redhat-00006//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
	at org.wildfly.extension.undertow@8.1.0.Beta-redhat-00006//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
	at org.wildfly.extension.undertow@8.1.0.Beta-redhat-00006//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)
	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)
	at io.undertow.core@2.3.18.SP1-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)
	at io.undertow.core@2.3.18.SP1-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:896)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at org.jboss.xnio@3.8.16.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:840)

      The client is just slightly affected because we observe the following two errors on the client side (complete log in jmeter-logs.zip )

      2025-03-25 23:29:08,589 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Thread Group 1-907: Serial 126: Error during request execution: statusCode: 500: Invalid HTTP response code from url http://10.0.197.28:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
java.lang.IllegalStateException: statusCode: 500: Invalid HTTP response code from url http://10.0.197.28:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
	at org.jboss.eapqe.clustering.jmeter.ClusteringHTTPRequestSampler.runTest(ClusteringHTTPRequestSampler.java:197) [clustering-jmeter-samplers.jar:?]
	at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) [ApacheJMeter_core.jar:5.6.3]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2025-03-25 23:29:08,594 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Thread Group 1-1439: Serial 122: Error during request execution: statusCode: 500: Invalid HTTP response code from url http://10.0.197.28:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
java.lang.IllegalStateException: statusCode: 500: Invalid HTTP response code from url http://10.0.197.28:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
	at org.jboss.eapqe.clustering.jmeter.ClusteringHTTPRequestSampler.runTest(ClusteringHTTPRequestSampler.java:197) [clustering-jmeter-samplers.jar:?]
	at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) [ApacheJMeter_core.jar:5.6.3]
	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) [ApacheJMeter_core.jar:5.6.3]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

              pferraro@redhat.com Paul Ferraro
              tborgato@redhat.com Tommaso Borgato
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: