This is with JGroups 2.6.1 on Linux 2.6.18, and a cluster of 10-20 nodes with several channels on each node.

Everything is running smoothly, and then we kill the coordinator (of one or more channels), plus the next few nodes in the view, all at about the same time. SUSPECTs happen, and then the highest-ranking remaining node decides to become coordinator. It tries to mcast a new view, but never gets any view ACKs - so nobody else sees the updated view and the channel remains basically unusable. This only happens sometimes, and usually on just some of several channels - even though they share the same stack config.

Enabling TRACE logging revealed that ENCRYPT.down() is queuing the new view - we use dynamically-generated keys and apparently the view changes have flipped queue_down to 'true'. We see this, over and over:

2008-03-02 10:54:47,970 [TRACE] GMS mcasting view

{[10.193.48.119:33274|693] [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645]}

(4 mbrs)
2008-03-02 10:54:47,970 [TRACE] ENCRYPT queueing down message as no session key established[dst: <null>, src: <null> (1 headers), size=0 bytes]
2008-03-02 10:54:49,972 [WARN] GMS failed to collect all ACKs (4) for view [10.193.48.119:33274|693] [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645] after 2000ms, missing ACKs from [10.193.48.119:33274, 10.193.48.115:52962, 10.192.226.30:45130, 10.193.66.111:43645] (received=[]), local_addr=10.193.48.119:33274