-
Bug
-
Resolution: Done
-
Critical
-
30.0.0.Beta1, 30.0.0.Final, 31.0.0.Beta1
-
None
-
-
Regression
-
---
-
Reported by QE
We hit this error when running a 4 nodes cluster where we deploy an application containing a stateful session bean; data is replicated across the cluster;
We start the cluster and then run ~4000 ejb clients from yet another node;
After some time and after having restarted all nodes at least once, the following error pops out on one or more nodes:
2023-10-27 11:15:37,894 ERROR [org.xnio.listener] (default I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$1.createBuffer(ByteBufferPool.java:514) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$2.createBuffer(ByteBufferPool.java:535) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780) at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161) at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.RemoteConnection.allocate(RemoteConnection.java:80) at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.ServerConnectionOpenListener.handleEvent(ServerConnectionOpenListener.java:95) at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:270) at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:244) at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:163) at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:160) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783) at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316) at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783) at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316) at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.jboss.xnio@3.8.11.Final//org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:132) at org.jboss.xnio@3.8.11.Final//org.xnio.Connection.close(Connection.java:142) at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection.close(AbstractServerConnection.java:159) at org.jboss.xnio@3.8.11.Final//org.xnio.IoUtils.safeClose(IoUtils.java:152) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:772) at io.undertow.core@2.3.10.Final//io.undertow.conduits.HeadStreamSinkConduit.flush(HeadStreamSinkConduit.java:131) at org.jboss.xnio@3.8.11.Final//org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) at io.undertow.core@2.3.10.Final//io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1813) at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783) at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:206) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http2.Http2UpgradeHandler.handleRequest(Http2UpgradeHandler.java:101) at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:62) at io.undertow.core@2.3.10.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100) at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178) at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:624) at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:491)
heap info of the WF instance emitting the error:
[hudson@tborgato-clustering-a-2 tmp]$ jps -lv 98419 /tmp/wildfly/jboss-modules.jar -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAMED --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/com.sun.net.ssl.internal.ssl=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED -Dorg.jboss 99151 jdk.jcmd/sun.tools.jps.Jps -Dapplication.home=/qa/tools/opt/x86_64/jdk-11.0.19 -Xms8m -Djdk.module.main=jdk.jcmd 98926 check-leak-0.11.jar [hudson@tborgato-clustering-a-2 tmp]$ jcmd 98419 GC.heap_info 98419: garbage-first heap total 462848K, used 276038K [0x00000000e0000000, 0x0000000100000000) region size 1024K, 0 young (0K), 0 survivors (0K) Metaspace used 117302K, capacity 130315K, committed 132608K, reserved 366592K class space used 15524K, capacity 19945K, committed 20736K, reserved 253952K [hudson@tborgato-clustering-a-2 tmp]$ jstat -gc 98419 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT CGC CGCT GCT 0.0 0.0 0.0 0.0 114688.0 6144.0 356352.0 277806.4 132864.0 117345.6 20736.0 15525.4 1506 10.827 333 185.297 320 5.613 201.737
Find attached:
- server.zip : logs of the WF instance emitting the error
- check-leak.zip : report created with check-leak of the WF instance emitting the error (io.undertow.conduits.ReadTimeoutStreamSourceConduit$1 is one of the first domain classes shown: )
- is caused by
-
REM3-404 Remoting connections closed during greetings exchange after HTTP upgrade are not properly cleaned after closed
- Closed
-
UNDERTOW-2336 CVE-2024-1635 At Http upgrade to remoting, WriteTimeoutStreamSinkConduit leaks connections if RemotingConnection is closed by Remoting ServerConnectionOpenListener
- Closed
-
XNIO-427 ClosedChannelException when NioSocketConduit.handleReady invokes write listener after read listener closes connection
- Closed
- is incorporated by
-
WFLY-19058 Upgrade WildFly Core to 23.0.3.Final
- Closed
-
WFLY-19076 Upgrade WildFly Core to 24.0.0.Beta2
- Closed
- relates to
-
JBEAP-26171 (8.0.z) OOM Error after node restart, in a 4 nodes cluster
- Closed
-
JBEAP-26462 (7.4.z) OOM Error after node restart, in a 4 nodes cluster
- Closed