17:31:01,970 ERROR [UDP] failed handling incoming message
java.util.ConcurrentModificationException
at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:365)
at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:376)
at java.util.AbstractCollection.containsAll(AbstractCollection.java:285)
at java.util.AbstractSet.equals(AbstractSet.java:76)
at org.jgroups.protocols.pbcast.STABLE.allVotesReceived(STABLE.java:428)
at org.jgroups.protocols.pbcast.STABLE.addVote(STABLE.java:423)
at org.jgroups.protocols.pbcast.STABLE.handleStableMessage(STABLE.java:554)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:238)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:299)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:842)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:705)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:167)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:308)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:263)
at org.jgroups.protocols.PING.up(PING.java:270)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
at org.jgroups.protocols.TP.access$100(TP.java:49)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

Here are the problematic lines in STABLE.java:

427: private boolean allVotesReceived(Set<Address> votes)

{ 428: return votes.equals(mbrs); // compares identity, size and element-wise (if needed) 429: }

Internally, the Set.equals(...) method will iterate over the mbrs set.
The concurrent update is coming from the following code in STABLE.java:

330: private void handleViewChange(View v) {
331: Vector<Address> tmp=v.getMembers();
332: synchronized(mbrs)

{ 333: mbrs.clear(); 334: mbrs.addAll(tmp); 335: }

There are several ways fix this, including, but not limited to:

• Wrap line 428 in a synchronized(mbrs) block

• Make mbrs volatile instead of final, and create a new LinkedHashSet in handleViewChange(...)
  i.e. Replace 332-335 with:
  mbrs = new LinkedHashSet<Address>(tmp);

• Make mbrs a CopyOnWriteArraySet<Address>