If the domain master is up but not responsive (such as an OOME) and results in repeating connection retries from slaves, then those connections leak each retry and build up. With SSL, those connections can tie up larger direct buffers and this behavior has led to a direct buffer OOME.

ProtocolConnectionUtils.connectSync calls remoting to connect and receives an IoFuture<Connection>, which it then waits on for, by default, 5 seconds. If the future completes or fails, it's fine. But if it times out connectSync throws an exception without first cancelling the future. This means remoting has no idea the connection isn't going to be used and if the connection ends up getting established it leaks via the chain WorkerThread -> selector -> table of SelectorKeys maintained by Selector -> NioSocketConduit -> NioSocketStreamConnection.

This has the potential to leak the connection on the other side as well, since if the connection was successfully established but never used or closed down by the initiating side it can just sit there.

One place where this can be more problematic is a remote HostController re-connecting to the DomainController after the HC successfully started but then the connection was lost. The HC will loop continually trying to connect. If this problem keeps happening the leaked connection objects will accumulate.