Critical Topology:
1 loadbalancer
2 workers
Test scenario:
1. spin up server instances, loadbalancer uses standalone-load-balancer.xml, workers standalone-ha.xml
2. configure workers
jvmRoute is either worker-1 or worker-2
//system-property=jboss.mod_cluster.jvmRoute:add(value=\"" + jvmRoute + "\") /subsystem=modcluster/proxy=default:write-attribute(name=advertise,value=false) /socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=proxy1:add(host=\"" + balancerIP + "\", port=\"" + balancerPort + "\") /subsystem=modcluster/proxy=default:list-add(name=proxies,value=proxy1) reload
3. deploy
Deployment is a very simple application printing worker id: https://gitlab.cee.redhat.com/jbossqe-eap/clouds-ts/-/blob/master/shared-tests/src/main/java/org/wildfly/test/hyperscaler/dynamicBalancer/deployment/ModclusterLoadbalancerResource.java
Deployment operation fails intermittently with following exception:
2025-12-16 12:25:37,042 ERROR [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0055: Caught exception during boot: java.util.concurrent.CompletionException: java.util.ConcurrentModificationException at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:413) at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2118) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.ServerService.finishBoot(ServerService.java:446) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.ServerService.boot(ServerService.java:396) at org.jboss.as.controller@27.1.2.Final-redhat-00002//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:358) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.util.ConcurrentModificationException at java.base/java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) at java.base/java.util.HashMap$KeyIterator.next(HashMap.java:1620) at org.wildfly.mod_cluster.undertow@8.1.1.GA-redhat-00006//org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.resume(UndertowEventHandlerAdapterService.java:196) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.ServerActivity.resume(ServerActivity.java:112) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:202) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:165) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:182) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:165) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:182) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:165) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:182) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:165) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:182) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:165) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController$1.accept(SuspendController.java:182) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController.phaseStage(SuspendController.java:212) at org.jboss.as.server@27.1.2.Final-redhat-00002//org.jboss.as.server.suspend.SuspendController.resume(SuspendController.java:136) ... 4 more
Logs and configurations are attached
I hit this on AWS EC2 instances. I suspect this plays a role since instances are not on the same machine and they run in parallel. From time to time deployment is successful when I pause the debugger before the deploy operation when I think I give the servers more time. But when I let it run without a pause I have no problem hitting it.
If there are troubles with reproducing the issue locally, I can assist with deploying to AWS env.
I don't know if this is a regression to 8.1.0.GA, we did not have a test EC2 image for 8.1.0.GA.