Major A customer using EAP 7.1.0 has experienced an OOM error since 7.1.3 + jbeap-15157 was applied.
[[0m[[31m06:20:42,344 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57)
at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55)
at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:147)
at io.undertow.server.XnioByteBufferPool.allocate(XnioByteBufferPool.java:53)
at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:147)
at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:148)
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:92)
at io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:51)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.nio.QueuedNioTcpServer$1.run(QueuedNioTcpServer.java:131)
at org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:479)
The OOM error occurred only in the production system and in two days after EAP is started.
The production system was deployed the same applications on EAP 7.1.0 instances and two EAP 7.1.3 instances, but only in 7.1.3 instances had an OOM error.
Currently, they have uninstalled all version 7.1.3 and there are no problems with version 7.1.0.
They uploaded three heapdumps. The heapdumps and the logs above were generated at different times.
1. Heapdump for 7.1.3 where OOM occurred : 713_OOM_ep2pd003_dump.hprof
2. Heapdump for 7.1.3 without OOM : 713_NoOOM_ep2pd003_dump.hprof
3. Heapdump for 7.1.0 normal : 710_ep2pd003_dump.hprof
Some of the odd things in the above heapdumps:
1) Heapdumps for 7.1.3 show that the number of instances of DirectByteBuffer and the size of retained memory are larger than those of 7.1.0.
// 7.1.3 with OOM
Class Name | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------
java.nio.DirectByteBuffer| 16,007 | 1,024,448 | >= 1,024,464
------------------------------------------------------------------
// 7.1.3 w/o OOM
Class Name | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------
java.nio.DirectByteBuffer| 62,364 | 3,991,296 | >= 4,008,160
------------------------------------------------------------------
//7.1.0
Class Name | Objects | Shallow Heap | Retained Heap
------------------------------------------------------------------
java.nio.DirectByteBuffer| 3,602 | 230,528 | >= 230,544
------------------------------------------------------------------
2) In the first heapdump(713_OOM_ep2pd003_dump.hprof), in the merge_shortest_paths of "DirectByteBuffer", it shows that the default I / O-5 thread has large retained heap.
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- org.xnio.nio.WorkerThread @ 0x7206c9928 default I/O-5 Thread | 2 | 152 | 128 | 1,103,480 '- <Java Local>, selector sun.nio.ch.EPollSelectorImpl @ 0x722a588f8 | 2 | 72 | 128 | 131,624 '- fdToKey java.util.HashMap @ 0x722a57c00 | 2 | 48 | 128 | 31,664 '- table java.util.HashMap$Node[1024] @ 0x6d3239f40 | 2 | 4,112 | 128 | 31,616 '- [128] java.util.HashMap$Node @ 0x70d4047f8 | 2 | 32 | 128 | 48 '- value sun.nio.ch.SelectionKeyImpl @ 0x70d404828 | 2 | 40 | 128 | 8,784 '- attachment org.xnio.nio.NioSocketConduit @ 0x70d404850 | 2 | 64 | 128 | 8,208 '- connection org.xnio.nio.NioSocketStreamConnection @ 0x70d404aa8 | 2 | 40 | 128 | 40 '- closeListener io.undertow.server.AbstractServerConnection$CloseSetter @ 0x70d4059f0| 2 | 24 | 128 | 24 '- this$0 io.undertow.server.protocol.http.HttpServerConnection @ 0x70d404c78 | 2 | 96 | 128 | 2,480 '- extraBytes io.undertow.server.XnioByteBufferPool$1 @ 0x70d405a40 | 2 | 24 | 128 | 200 '- val$buf org.xnio.ByteBufferSlicePool$PooledByteBuffer @ 0x70d406278 | 2 | 24 | 128 | 176 |- buffer java.nio.DirectByteBuffer @ 0x70d4062a8 | 1 | 64 | 64 | 64 |- region org.xnio.ByteBufferSlicePool$Slice @ 0x70d406290 | 1 | 24 | 64 | 24 | '- parent java.nio.DirectByteBuffer @ 0x70d4062e8 | 1 | 64 | 64 | 64 '- Total: 2 entries | | | | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Environment
-----------
RHEL 7.3
Oralce JDK 8 update 143
I'll provide more information as soon as I can get.
