At some point, TCP_NIO2 starts dropping messages, at either the send or receive direction (still need to find out). This shows itself when e.g. a JOIN fails: the joiner sends a lot of JOIN-REQ messages and gets JOIN-RSP msgs from the coord, but doesn't receive them.
If GMS.max_join_attempts="0", then this goes on forever, or until the joiner doesn't discover any nodes anymore.
probe.sh jmx=UNICAST3.printConnections shows that messages are sent but no acks are received.
Clearing the connection table in TCP_NIO2 fixes the problem: probe.sh op=TCP_NIO2.clearConnections.
This shows that the issue is indeed in TCP_NIO2. Also, TCP and UDP work.