I built a game server, using Undertow websockets implemented through JSR356.

The game server has to loops, using ScheduledExecutorService.scheduleAtFixedRate, the game loop, and the data loop, which runs 20 times per second sending data of the world to all clients using getAsyncRemote().sendBinary(ByteBuffer).

Everything works right for a while, until at some moment, memory starts to grow, and some minutes later everything becomes unstable, people stop receiving packets and I have no other way than to restart the server.

I hooked jvisualvm using JMX and took some heap dumps, I attach the pictures of 3 of them, in temporal order.

In resume, the following classes grow to hundreds of thousands:
org.xnio.ChannelListener$SimpleSetter
java.nio.HeapByteBuffer
io.undertow.util.ImmediatePooledByteBuffer
io.undertow.websockets.core.WebSocket$2
io.undertow.websockets.core.WebSocket$1
io.undertow.websockets.jsr.SendResultFuture

You can download all the heap dumps to open with jvisualvm from here: https://mega.nz/#!4ckVWaAI!RQoqHdtAtwrah2Q_SBd2870AoNj3u78ry46h-B1aMXY