We are using JGroups Version 2.6.7 GA.

When a cluster spans at least two subnets, cluster members become disconnected and the only way to get them to reconnect to the cluster is to bring all of the processes down and bring them back up at the same time.

Bouncing one box at a time does not work. We have not seen this issue at all when all of the cluster members are in the same subnet.

Also happened in JGroups version 2.3 SP1.

This is an intermittent problem. Customers can normally run for several days without issue. Then the cluster will split and never fix itself. The only solution is to bring down all boxes.

The configuration that is active when the situation occurs is:

TCP(start_port=7801;external_addr=192.168.218.62):
TCPPING(initial_hosts=192.168.218.62[7801],192.168.128.62[7801];port_range=2;timeout=3500;num_initial_members=2;up_thread=true;down_thread=true):
MERGE2(min_interval=5000;max_interval=10000):
FD_SOCK(bind_addr=192.168.218.62):
FD(shun=true;timeout=2500;max_tries=5;up_thread=true;down_thread=true):
VERIFY_SUSPECT(timeout=2000;down_thread=false;up_thread=false):
pbcast.NAKACK(down_thread=true;up_thread=true;gc_lag=100;retransmit_timeout=3000):
pbcast.STABLE(desired_avg_gossip=20000;down_thread=false;up_thread=false):
pbcast.STATE_TRANSFER(down_thread=false;up_thread=false):
pbcast.GMS(join_timeout=60000;join_retry_timeout=60000;shun=true;print_local_addr=true;down_thread=true;up_thread=true)

I will be posting logs from the customer's site shortly.