Scenario: we have a 4 nodes cluster configured to use the TPC stack and the default cache configuration for web and ejb;

On each node we deploy clusterbench-ee10.ear and the context path /clusterbench/ejbservlet is invoked by ~600 clients per node every 4 seconds (= EJB-SERVLET scenario);
The context path /clusterbench/ejbservlet provides and integer value which is incremented at every invocation and stored in the ejb cache to be distributed across the cluster (the servlet that is invoked uses a Stateful Session Bean);

Fail-over is introduced by shutting down and re-starting each node in a sequence;

The series of events that lead to the error is the following:

• node1, node2, node3, node4 are started and form a cluster
• clusterbench-ee10.ear is deployed to every node
• node1 is stopped and then re-stated
• node2 is stopped
• node3 emits the following error:

  2023-10-12 08:02:45,980 ERROR [io.undertow.request] (default task-15) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.module.web.context.beanstore.http.AbstractSessionBeanStore.getLockStore(AbstractSessionBeanStore.java:125)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.contexts.beanstore.AttributeBeanStore.lock(AttributeBeanStore.java:256)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:90)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.contexts.PassivatingContextWrapper$AbstractPassivatingContextWrapper.get(PassivatingContextWrapper.java:84)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.get(ContextualInstanceStrategy.java:177)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:102)
  	at org.jboss.weld.core@5.1.1.SP2-redhat-00001//org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:131)
  	at deployment.clusterbench-ee10.ear.clusterbench-ee10-ejb.jar//org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source)
  	at deployment.clusterbench-ee10.ear.clusterbench-ee10-web.war//org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:41)
  	at jakarta.servlet.api@6.0.0.redhat-00001//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
  	at jakarta.servlet.api@6.0.0.redhat-00001//jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  	at org.wildfly.security.elytron-web.undertow-server@4.0.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.ElytronRunAsHandler.lambda$handleRequest$1(ElytronRunAsHandler.java:68)
  	at org.wildfly.security.elytron-base@2.2.2.Final-redhat-00001//org.wildfly.security.auth.server.FlexibleIdentityAssociation.runAsFunctionEx(FlexibleIdentityAssociation.java:103)
  	at org.wildfly.security.elytron-base@2.2.2.Final-redhat-00001//org.wildfly.security.auth.server.Scoped.runAsFunctionEx(Scoped.java:161)
  	at org.wildfly.security.elytron-base@2.2.2.Final-redhat-00001//org.wildfly.security.auth.server.Scoped.runAs(Scoped.java:73)
  	at org.wildfly.security.elytron-web.undertow-server@4.0.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.ElytronRunAsHandler.handleRequest(ElytronRunAsHandler.java:67)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
  	at org.wildfly.security.elytron-web.undertow-server-servlet@4.0.0.Final-redhat-00001//org.wildfly.elytron.web.undertow.server.servlet.CleanUpHandler.handleRequest(CleanUpHandler.java:38)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:276)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
  	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20231010//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)
  	at io.undertow.servlet@2.3.7.SP2-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
  	at io.undertow.core@2.3.7.SP2-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
  	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
  	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
  	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
  	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
  	at org.jboss.xnio@3.8.9.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
  	at java.base/java.lang.Thread.run(Thread.java:834)
  

At the same time we have the following errors on the client:

2023-10-12 08:02:45,357 WARN o.j.e.c.j.ClusteringHTTPRequestSampler: Thread Group 1-1816: Serial 84: Session ID changed: JSessionId{sessionId='ohgBEKzkKbX5QxP9qYGfkN83Y3dI2QlkXBMFt8b4', jvmRoutes=[wildfly2]} -> JSessionId{sessionId='DLSpQqMhEXNS0XbqlnoWc3yuKUaAJj_oeKRruFDv', jvmRoutes=[wildfly3]}
2023-10-12 08:02:45,358 ERROR o.j.e.c.j.ClusteringHTTPRequestSampler: Invalid serial: Expected 84, received 0 (previous: JSessionId{sessionId='ohgBEKzkKbX5QxP9qYGfkN83Y3dI2QlkXBMFt8b4', jvmRoutes=[wildfly2]} current: JSessionId{sessionId='DLSpQqMhEXNS0XbqlnoWc3yuKUaAJj_oeKRruFDv', jvmRoutes=[wildfly3]}) 

Overall client fail rate is till very low;

Complete logs in wildfly-ejb-ejbservlet-shutdown-dist.zip;

The error is intermittent: it showed up 4 times in 15 runs;