-
Bug
-
Resolution: Done
-
Critical
-
30.0.0.Beta1, 30.0.0.Final, 31.0.0.Beta1
-
None
-
Download and unzip JMeter
Run the following on e.g. your laptop or some node you will be using as client:
echo '====================================================' echo 'prepare jmeter...' echo '====================================================' wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.2.zip unzip -q apache-jmeter-5.6.2.zip
Prepare a 4 nodes cluster
You will need another 4 hosts for WF instances;
We used 4 VMs with the following properties:
- RAM: 4GB
- VCPUs: 2 VCPU
- Disk: 40GB
Configure each node as in the following:
echo '====================================================' echo 'prepare wildlfy...' echo '====================================================' wget https://github.com/wildfly/wildfly/releases/download/30.0.0.Final/wildfly-30.0.0.Final.zip export WILDFLY_ZIP=wildfly-30.0.0.Final.zip export WILDFLY_DIR=wildfly-30.0.0.Final rm -rdf wildfly unzip -q $WILDFLY_ZIP mv $WILDFLY_DIR wildfly # create user for the ejb clients ./wildfly/bin/add-user.sh -u joe -p secret-Passw0rd -a
Deploy clusterbench on every node of the cluster
Use the attached clusterbench-ee10.ear
(or build it from https://github.com/clusterbench/clusterbench):cp clusterbench-ee10.ear ./wildfly/standalone/deployments/
Start the 4 WF instances
Node1:
NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly1
Node2:
NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly2
Node3:
NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly3
Node4:
NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly4
Start the JMeter client
Find the IPs of the nodes where you run WF and store them in the following shell variables:
NODE1_IP=... NODE2_IP=... NODE3_IP=... NODE4_IP=...
Use the attached clustering-jmeter-samplers-jar-with-dependencies.jar
which contains the Java class performing the remote EJB invocation (you can also build it yourself from this repo: https://github.com/tommaso-borgato/clustering-jmeter-samplers) and the TestPlanEJB.jmx JMeter test plan:./apache-jmeter-5.6.2/bin/jmeter -n \ -t TestPlanEJB.jmx \ -Jjmeter.save.saveservice.output_format=csv \ -Jjmeter.save.saveservice.default_delimiter="," \ -Jjmeter.save.saveservice.autoflush=true \ -l jmeter_results-perf.csv \ -Jhost=$NODE1_IP,$NODE2_IP,$NODE3_IP,$NODE4_IP \ -Jport=8080,8080,8080,8080 \ -Jpath=/clusterbench/session \ -Jusername=joe -Jpassword=secret-Passw0rd -Jusers=4000 -Jrampup=60 -Jremote.prog=0 \ -Jjmeter.save.saveservice.timestamp_format='yyyy/M/dd HH:mm:ss' \ -Lorg.jboss.eapqe.clustering.jmeter=DEBUG \ -Juser.classpath=clustering-jmeter-samplers-jar-with-dependencies.jar
Restart all nodes at least once
After the 4 nodes are up and the jmeter client successfully started the 4000 clients, stop (CTRL+C from terminal) and restart each node with a 3 minutes interval: this means you leave the cluster running for 5 minutes, the you stop one node, wait 3 minutes, restart the node, wait 3 minutes, stop another node... then wait some time and, eventually, you will be able to observe the error;
NOTE: the error doesn't show up if you run the test locally (4 nodes and client on the same machine)
NOTE: this error hasn't surfaced in EAP yet
Download and unzip JMeter Run the following on e.g. your laptop or some node you will be using as client: echo '====================================================' echo 'prepare jmeter...' echo '====================================================' wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.2.zip unzip -q apache-jmeter-5.6.2.zip Prepare a 4 nodes cluster You will need another 4 hosts for WF instances; We used 4 VMs with the following properties: RAM: 4GB VCPUs: 2 VCPU Disk: 40GB Configure each node as in the following: echo '====================================================' echo 'prepare wildlfy...' echo '====================================================' wget https://github.com/wildfly/wildfly/releases/download/30.0.0.Final/wildfly-30.0.0.Final.zip export WILDFLY_ZIP=wildfly-30.0.0.Final.zip export WILDFLY_DIR=wildfly-30.0.0.Final rm -rdf wildfly unzip -q $WILDFLY_ZIP mv $WILDFLY_DIR wildfly # create user for the ejb clients ./wildfly/bin/add-user.sh -u joe -p secret-Passw0rd -a Deploy clusterbench on every node of the cluster Use the attached clusterbench-ee10.ear (or build it from https://github.com/clusterbench/clusterbench): cp clusterbench-ee10.ear ./wildfly/standalone/deployments/ Start the 4 WF instances Node1: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly1 Node2: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly2 Node3: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly3 Node4: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly4 Start the JMeter client Find the IPs of the nodes where you run WF and store them in the following shell variables: NODE1_IP=... NODE2_IP=... NODE3_IP=... NODE4_IP=... Use the attached clustering-jmeter-samplers-jar-with-dependencies.jar which contains the Java class performing the remote EJB invocation (you can also build it yourself from this repo: https://github.com/tommaso-borgato/clustering-jmeter-samplers ) and the TestPlanEJB.jmx JMeter test plan: ./apache-jmeter-5.6.2/bin/jmeter -n \ -t TestPlanEJB.jmx \ -Jjmeter.save.saveservice.output_format=csv \ -Jjmeter.save.saveservice.default_delimiter="," \ -Jjmeter.save.saveservice.autoflush=true \ -l jmeter_results-perf.csv \ -Jhost=$NODE1_IP,$NODE2_IP,$NODE3_IP,$NODE4_IP \ -Jport=8080,8080,8080,8080 \ -Jpath=/clusterbench/session \ -Jusername=joe -Jpassword=secret-Passw0rd -Jusers=4000 -Jrampup=60 -Jremote.prog=0 \ -Jjmeter.save.saveservice.timestamp_format='yyyy/M/dd HH:mm:ss' \ -Lorg.jboss.eapqe.clustering.jmeter=DEBUG \ -Juser.classpath=clustering-jmeter-samplers-jar-with-dependencies.jar Restart all nodes at least once After the 4 nodes are up and the jmeter client successfully started the 4000 clients, stop (CTRL+C from terminal) and restart each node with a 3 minutes interval: this means you leave the cluster running for 5 minutes, the you stop one node, wait 3 minutes, restart the node, wait 3 minutes, stop another node... then wait some time and, eventually, you will be able to observe the error; NOTE: the error doesn't show up if you run the test locally (4 nodes and client on the same machine) NOTE: this error hasn't surfaced in EAP yet -
Regression
-
---
-
Reported by QE
We hit this error when running a 4 nodes cluster where we deploy an application containing a stateful session bean; data is replicated across the cluster;
We start the cluster and then run ~4000 ejb clients from yet another node;
After some time and after having restarted all nodes at least once, the following error pops out on one or more nodes:
2023-10-27 11:15:37,894 ERROR [org.xnio.listener] (default I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$1.createBuffer(ByteBufferPool.java:514)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$2.createBuffer(ByteBufferPool.java:535)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780)
at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161)
at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.RemoteConnection.allocate(RemoteConnection.java:80)
at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.ServerConnectionOpenListener.handleEvent(ServerConnectionOpenListener.java:95)
at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:270)
at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:244)
at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:163)
at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:160)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316)
at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316)
at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio@3.8.11.Final//org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:132)
at org.jboss.xnio@3.8.11.Final//org.xnio.Connection.close(Connection.java:142)
at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection.close(AbstractServerConnection.java:159)
at org.jboss.xnio@3.8.11.Final//org.xnio.IoUtils.safeClose(IoUtils.java:152)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:772)
at io.undertow.core@2.3.10.Final//io.undertow.conduits.HeadStreamSinkConduit.flush(HeadStreamSinkConduit.java:131)
at org.jboss.xnio@3.8.11.Final//org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
at io.undertow.core@2.3.10.Final//io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1813)
at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:206)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http2.Http2UpgradeHandler.handleRequest(Http2UpgradeHandler.java:101)
at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:62)
at io.undertow.core@2.3.10.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178)
at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:624)
at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:491)
heap info of the WF instance emitting the error:
[hudson@tborgato-clustering-a-2 tmp]$ jps -lv 98419 /tmp/wildfly/jboss-modules.jar -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAMED --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/com.sun.net.ssl.internal.ssl=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED -Dorg.jboss 99151 jdk.jcmd/sun.tools.jps.Jps -Dapplication.home=/qa/tools/opt/x86_64/jdk-11.0.19 -Xms8m -Djdk.module.main=jdk.jcmd 98926 check-leak-0.11.jar [hudson@tborgato-clustering-a-2 tmp]$ jcmd 98419 GC.heap_info 98419: garbage-first heap total 462848K, used 276038K [0x00000000e0000000, 0x0000000100000000) region size 1024K, 0 young (0K), 0 survivors (0K) Metaspace used 117302K, capacity 130315K, committed 132608K, reserved 366592K class space used 15524K, capacity 19945K, committed 20736K, reserved 253952K [hudson@tborgato-clustering-a-2 tmp]$ jstat -gc 98419 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT CGC CGCT GCT 0.0 0.0 0.0 0.0 114688.0 6144.0 356352.0 277806.4 132864.0 117345.6 20736.0 15525.4 1506 10.827 333 185.297 320 5.613 201.737
Find attached:
- server.zip : logs of the WF instance emitting the error
- check-leak.zip : report created with check-leak of the WF instance emitting the error (io.undertow.conduits.ReadTimeoutStreamSourceConduit$1 is one of the first domain classes shown:
)
- is caused by
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
- is incorporated by
-
WFLY-19058 Upgrade WildFly Core to 23.0.3.Final
-
- Closed
-
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
-
- Closed
-
(2 relates to)