Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-30511

(8.1.z) [CLUSTERING] Intermittent "HTTP/1.1 500 Internal Server Error" when server is shut down

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • None
    • 8.1.0.GA-CR7
    • Clustering
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • +

      Scenario: 4 nodes cluster where we deploy clusterbench and use EJB+SERVLET data replication;

      We spotted the following issue after shsut-down was initiated on one of the 4 nodes (complete logs in wildfly.zip ):

      2025-07-09 00:08:24,495 INFO  [org.jboss.as.server] (management-handler-thread - 3) WFLYSRV0211: Suspending server with 15000 ms timeout.
      2025-07-09 00:08:24,497 INFO  [org.jboss.as.ejb3] (management-handler-thread - 3) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
      2025-07-09 00:08:24,514 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
      2025-07-09 00:08:24,538 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-5) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
      2025-07-09 00:08:24,542 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 83) [Context=org.infinispan.CONFIG] ISPN100008: Updating cache members list [wildfly4, wildfly1, wildfly2], topology id 36
      2025-07-09 00:08:24,543 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 83) [Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [wildfly4, wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 37
      2025-07-09 00:08:24,546 INFO  [org.infinispan.CONTAINER] (ServerService Thread Pool -- 83) ISPN000390: Persisted state, version=15.0.14.Final-redhat-00002 timestamp=2025-07-09T00:08:24.545901514Z
      2025-07-09 00:08:24,551 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 83) Stopped server cache container
      2025-07-09 00:08:24,560 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 86) [Context=org.infinispan.CONFIG] ISPN100008: Updating cache members list [wildfly4, wildfly1, wildfly2], topology id 36
      2025-07-09 00:08:24,563 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 86) [Context=org.infinispan.CONFIG] ISPN100002: Starting rebalance with members [wildfly4, wildfly1, wildfly2], phase READ_OLD_WRITE_ALL, topology id 37
      2025-07-09 00:08:24,565 INFO  [org.infinispan.CONTAINER] (ServerService Thread Pool -- 86) ISPN000390: Persisted state, version=15.0.14.Final-redhat-00002 timestamp=2025-07-09T00:08:24.565591550Z
      2025-07-09 00:08:24,566 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 86) Stopped hibernate cache container
      2025-07-09 00:08:24,571 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 83) WFLYUT0022: Unregistered web context: '/clusterbench' from server 'default-server'
      2025-07-09 00:08:24,575 ERROR [io.undertow.servlet.request] (default task-43) UT015005: Error invoking method requestInitialized on listener class org.jboss.weld.module.web.servlet.WeldInitialListener: java.lang.IllegalStateException: WFLYCLWEBUT0003: Session manager was stopped
      	at org.wildfly.clustering.web.undertow@8.1.0.GA-redhat-00009//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSessionCloseTask(DistributableSessionManager.java:103)
      	at org.wildfly.clustering.web.undertow@8.1.0.GA-redhat-00009//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:201)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:884)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:454)
      	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
      	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:266)
      	at org.jboss.weld.core@5.1.5.Final-redhat-00001//org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:151)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:263)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
      	at org.wildfly.extension.undertow@8.1.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
      	at org.wildfly.extension.undertow@8.1.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
      	at org.wildfly.extension.undertow@8.1.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
      	at org.wildfly.extension.undertow@8.1.0.GA-redhat-00009//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1421)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)
      	at io.undertow.servlet@2.3.18.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)
      	at io.undertow.core@2.3.18.SP1-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:395)
      	at io.undertow.core@2.3.18.SP1-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:896)
      	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
      	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at org.jboss.xnio@3.8.16.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
      	at java.base/java.lang.Thread.run(Thread.java:840)
      

      which, allegedly, caused an "HTTP/1.1 500 Internal Server Error" on the client:

      2025-07-09 00:08:24,578 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Thread Group 1-1160 Serial JSessionId{sessionId='XSMeekseRYfNS-zeLMKgocRfUcbcrLCJYTLqTEQy', jvmRoutes=[wildfly3]} JSESSIONID 124 Error during request execution: statusCode: 500: Invalid HTTP response code from url http://10.0.196.219:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
      java.lang.IllegalStateException: statusCode: 500: Invalid HTTP response code from url http://10.0.196.219:8080/clusterbench/ejbservlet: HTTP/1.1 500 Internal Server Error
      	at org.jboss.eapqe.clustering.jmeter.ClusteringHTTPRequestSampler.runTest(ClusteringHTTPRequestSampler.java:197) [clustering-jmeter-samplers.jar:?]
      	at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:197) [ApacheJMeter_java.jar:5.6.3]
      	at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) [ApacheJMeter_core.jar:5.6.3]
      	at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) [ApacheJMeter_core.jar:5.6.3]
      	at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) [ApacheJMeter_core.jar:5.6.3]
      	at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) [ApacheJMeter_core.jar:5.6.3]
      	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
      

      We spotted the issue only once and it didn't affect the client fail rate in a significant way, nevertheless, given the nature of the issue, we reported it here for analysis;

              pferraro@redhat.com Paul Ferraro
              tborgato@redhat.com Tommaso Borgato
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: