I implemented method org.jgroups.auth.AuthToken#authenticate(AuthToken token, Message msg) in my class and its body contained only one line: return false;
In this way authentication should be false and I should get SecurityException.
When I started joining of nodes together to form a cluster, instead of getting SecurityException, nodes formed 2 different clusters with the same name.
I am sure method was evaluated, since I tried to run it also with breakpoint, which was triggered during joining process.