Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
7.0.9.GA
-
None
Description
JBEAP 7.0.9,
- Facing the following ERROR:
2019-08-29 13:59:59,555 ERROR [org.xnio.listener] (default I/O-18) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) [rt.jar:1.8.0_221] at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.8.0_221] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) [rt.jar:1.8.0_221] at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:147) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.jboss.remoting3.remote.RemoteConnection.allocate(RemoteConnection.java:78) at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:242) at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:158) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:198) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:112) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1122) at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) [xnio-nio-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] at org.xnio.nio.WorkerThread.run(WorkerThread.java:571) [xnio-nio-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1] - Heap dump shows the direct byte usage from over 80,000 DirectByteBuffers and the majority from buffers sized at 16 mb each. I see these buffers cached under netty thread local WeakHashMap caches:
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.xnio.nio.WorkerThread @ 0x80510bd8 default I/O-13 Thread | 4,414 | 152 | 282,496 | 51,664 '- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x40079df28 | 4,414 | 24 | 282,496 | 36,248 '- table java.lang.ThreadLocal$ThreadLocalMap$Entry[64] @ 0xa973d858 | 4,414 | 272 | 282,496 | 36,224 '- [22] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x3f039fe68 | 4,414 | 32 | 282,496 | 27,264 '- value io.netty.util.internal.InternalThreadLocalMap @ 0x3f039fe88 | 4,414 | 128 | 282,496 | 27,232 '- indexedVariables java.lang.Object[32] @ 0x3f039ff08 | 4,414 | 144 | 282,496 | 26,712 |- [6] java.util.WeakHashMap @ 0x3f03a0468 | 4,171 | 48 | 266,944 | 20,208 | '- table java.util.WeakHashMap$Entry[1024] @ 0x3f03a0498 | 4,171 | 4,112 | 266,944 | 20,112 | |- [580] java.util.WeakHashMap$Entry @ 0x3f03a35f0 | 59 | 40 | 3,776 | 120 | | |- next java.util.WeakHashMap$Entry @ 0x3f03a3618 | 46 | 40 | 2,944 | 80 | | |- value io.netty.util.Recycler$WeakOrderQueue @ 0x8ed5ff58 | 13 | 32 | 832 | 2,680 | | | |- head io.netty.util.Recycler$WeakOrderQueue$Link @ 0x2d5bee4b0 16 | 10 | 32 | 640 | 1,952 | | | |- tail io.netty.util.Recycler$WeakOrderQueue$Link @ 0x3199882b0 7 | 3 | 32 | 192 | 664 | | | | '- elements io.netty.util.Recycler$DefaultHandle[16] @ 0x3199882d0 | 3 | 80 | 192 | 632 | | | | |- [1] io.netty.util.Recycler$DefaultHandle @ 0x3fc8eb570 | 1 | 32 | 64 | 184 | | | | | '- value io.netty.buffer.PooledUnsafeDirectByteBuf @ 0x3fc8eb590| 1 | 88 | 64 | 152 | | | | | '- tmpNioBuf java.nio.DirectByteBuffer @ 0x31b64e270 | 1 | 64 | 64 | 64 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- Tried to use "-XX:MaxDirectMemorySize" JVM option without success.
- Native memory tracking shows:
Diff1: Internal (reserved=1863514KB +52985KB, committed=1863514KB +52985KB) (malloc=1863482KB +52985KB #202054 +66983) (mmap: reserved=32KB, committed=32KB) Diff2: - Internal (reserved=43919126KB +42108597KB, committed=43919126KB +42108597KB) (malloc=43919094KB +42108597KB #661503 +526432) (mmap: reserved=32KB, committed=32KB) - Around 42GB of memory is occupied by NMT memory category ‘Internal’.
