Stack trace produced by NIO2 when group member process killed

This is more of a niggle, and possibly not one that is easily solved (or bothers many people).

In our installation, it's not uncommon for a process to be killed forceably. The NIO2 protocol always produces a stack trace on the log when this occurs, which gets fairly noisy for something that is a known occurence.

TBH, I'm not totally sure how to improve this. There is existing coding in there to reduce current noise (498, in NioConnector):

  catch(Exception ex) {
      if(!(ex instanceof SocketException || ex instanceof EOFException
        || ex instanceof ClosedChannelException))
          server.log.warn("failed handling message", ex);

The problem is that Java throws a rather unhelpful general IOException here, which skips the above tests:

21-11-24 08:48:14  WARN - failed handling message
java.io.IOException: An existing connection was forcibly closed by the remote host        at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)

We can work around this by setting the BaseServer log level to ERROR, but then we'll miss all warnings, which is not desirable either.

One thing that might help is if the above log message did not always produce the stacktrace e.g. always log the warning, and the message text from the exception. But then have a separate log call which only produces a stack trace if a finer grainer logging level is set e.g. trace/debug logging etc. That way, the warnings still appear for those who need them, and the stack traces are available if required by lower the log level.