Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-18254

NullPointerException during rebalance

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 29.0.0.Final
    • 29.0.0.Beta1
    • Clustering
    • None

      Scenario: we have a four nodes cluster and we trigger fail-over by failing the nodes one by one in a sequence (JVM Kill and EAP Shut-down);

      We spotted the following java.lang.NullPointerException errors in various phases of the re-balance:

      Before re-balance:

      2023-07-12 08:13:45,452 INFO  [org.infinispan.LIFECYCLE] (thread-23,ejb,wildfly4) [Context=clusterbench-ee10.ear.clusterbench-ee10-web.war] ISPN100002: Starting rebalance with members [wildfly2, wildfly3, wildfly4], phase TRANSITORY, topology id 9
2023-07-12 08:13:45,456 ERROR [io.undertow.servlet.request] (default task-15) UT015005: Error invoking method requestInitialized on listener class org.jboss.weld.module.web.servlet.WeldInitialListener: java.lang.NullPointerException
	at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-20230711//org.wildfly.clustering.server.infinispan.group.CacheGroup.createNode(CacheGroup.java:130)
	at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-20230711//org.wildfly.clustering.server.infinispan.group.CacheGroup.createNode(CacheGroup.java:64)
	at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-20230711//org.wildfly.clustering.ee.infinispan.PrimaryOwnerLocator.apply(PrimaryOwnerLocator.java:57)
	at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-20230711//org.wildfly.clustering.ee.infinispan.PrimaryOwnerLocator.apply(PrimaryOwnerLocator.java:38)
	at org.wildfly.clustering.web.infinispan@8.0.0.GA-redhat-20230711//org.wildfly.clustering.web.infinispan.routing.PrimaryOwnerRouteLocator.locate(PrimaryOwnerRouteLocator.java:55)
	at org.wildfly.clustering.web.undertow@8.0.0.GA-redhat-20230711//org.wildfly.clustering.web.undertow.routing.DistributableSessionIdentifierCodec.encode(DistributableSessionIdentifierCodec.java:45)
	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20230711//org.wildfly.extension.undertow.session.CodecSessionConfig.setSessionId(CodecSessionConfig.java:46)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl$ServletContextSessionConfig.setSessionId(ServletContextImpl.java:1257)
	at org.wildfly.clustering.web.undertow@8.0.0.GA-redhat-20230711//org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:235)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:884)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:454)
	at org.jboss.weld.core@5.1.1.Final-redhat-00001//org.jboss.weld.module.web.servlet.SessionHolder.requestInitialized(SessionHolder.java:47)
	at org.jboss.weld.core@5.1.1.Final-redhat-00001//org.jboss.weld.module.web.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:260)
	at org.jboss.weld.core@5.1.1.Final-redhat-00001//org.jboss.weld.module.web.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:146)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:263)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:132)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20230711//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20230711//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20230711//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
	at org.wildfly.extension.undertow@8.0.0.GA-redhat-20230711//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1430)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:256)
	at io.undertow.servlet@2.3.7.SP1-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:101)
	at io.undertow.core@2.3.7.SP1-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.core@2.3.7.SP1-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
	at org.jboss.xnio@3.8.9.Final-redhat-00001//org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:834)

      After re-balance:

      2023-07-12 08:12:45,148 INFO  [org.infinispan.LIFECYCLE] (thread-20,ejb,wildfly4) [Context=clusterbench-ee10.ear.clusterbench-ee10-web.war] ISPN100010: Finished rebalance with members [wildfly2, wildfly3, wildfly4], topology id 17
2023-07-12 08:12:45,150 INFO  [org.infinispan.LIFECYCLE] (non-blocking-thread--p8-t4) [Context=http-remoting-connector] ISPN100010: Finished rebalance with members [wildfly2, wildfly3, wildfly4], topology id 17
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) java.lang.NullPointerException
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.getPhysicalAddresses(FD_SOCK2.java:445)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.connectTo(FD_SOCK2.java:395)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.connectToNextPingDest(FD_SOCK2.java:376)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handle(FD_SOCK2.java:347)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handle(FD_SOCK2.java:31)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.ProcessingQueue.process(ProcessingQueue.java:55)
2023-07-12 08:14:39,828 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.ProcessingQueue.add(ProcessingQueue.java:35)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handleView(FD_SOCK2.java:364)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.down(FD_SOCK2.java:227)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.down(FailureDetection.java:149)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.down(VERIFY_SUSPECT2.java:84)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:640)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.down(UNICAST3.java:639)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:260)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.installView(GMS.java:676)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:66)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:112)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.handle(GMS.java:991)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:855)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:498)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:1032)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:965)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.handleMessageBatch(NAKACK2.java:937)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:768)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:314)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.RED.up(RED.java:119)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.TP.passBatchUp(TP.java:1209)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:286)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:275)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.wildfly.clustering.context@8.0.0.GA-redhat-20230711//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at org.wildfly.clustering.context@8.0.0.GA-redhat-20230711//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
2023-07-12 08:14:39,829 ERROR [stderr] (thread-22,ejb,wildfly4) 	at java.base/java.lang.Thread.run(Thread.java:834)

      2023-07-12 08:15:01,972 INFO  [org.infinispan.LIFECYCLE] (thread-12,ejb,wildfly1) [Context=clusterbench-ee10.ear.clusterbench-ee10-web.war] ISPN100010: Finished rebalance with members [wildfly3, wildfly4, wildfly1], topology id 26
2023-07-12 08:16:39,788 ERROR [stderr] (thread-60,ejb,wildfly1) java.lang.NullPointerException
2023-07-12 08:16:39,788 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.getPhysicalAddresses(FD_SOCK2.java:445)
2023-07-12 08:16:39,788 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.connectTo(FD_SOCK2.java:395)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.connectToNextPingDest(FD_SOCK2.java:376)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handle(FD_SOCK2.java:347)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handle(FD_SOCK2.java:31)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.ProcessingQueue.process(ProcessingQueue.java:55)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.ProcessingQueue.add(ProcessingQueue.java:35)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.handleView(FD_SOCK2.java:364)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.down(FD_SOCK2.java:227)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.down(FailureDetection.java:149)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Encrypt.down(Encrypt.java:127)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.ASYM_ENCRYPT.down(ASYM_ENCRYPT.java:123)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.down(VERIFY_SUSPECT2.java:84)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.down(NAKACK2.java:640)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.down(UNICAST3.java:639)
2023-07-12 08:16:39,789 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.stack.Protocol.down(Protocol.java:283)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:260)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.installView(GMS.java:676)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:66)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:112)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.handle(GMS.java:991)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:855)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:246)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.AUTH.up(AUTH.java:141)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:498)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:1032)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:965)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.handleMessageBatch(NAKACK2.java:937)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:768)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:119)
2023-07-12 08:16:39,790 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Encrypt.up(Encrypt.java:222)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.ASYM_ENCRYPT.up(ASYM_ENCRYPT.java:179)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:202)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:288)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:314)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.RED.up(RED.java:119)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.protocols.TP.passBatchUp(TP.java:1209)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:286)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:157)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.jgroups@5.2.14.Final-redhat-00001//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:275)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.wildfly.clustering.context@8.0.0.GA-redhat-20230711//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at org.wildfly.clustering.context@8.0.0.GA-redhat-20230711//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
2023-07-12 08:16:39,791 ERROR [stderr] (thread-60,ejb,wildfly1) 	at java.base/java.lang.Thread.run(Thread.java:834)

              pferraro@redhat.com Paul Ferraro
              pferraro@redhat.com Paul Ferraro
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: