19:22:54,187 DEBUG (testng-Test:[]) [Configurator] set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75 19:22:54,188 DEBUG (testng-Test:[]) [TCP_NIO2] thread pool min/max/keep-alive: 0/8/60000 use_fork_join=false, internal pool: 0/8/30000 (8 cores available) 19:22:54,277 DEBUG (testng-Test:[]) [GMS] address=J, cluster=ISPN, physical address=127.0.0.1:8600 19:22:54,278 TRACE (testng-Test:[]) [GMS] J: no members discovered after 1 ms: creating cluster as first member 19:22:54,278 DEBUG (testng-Test:[]) [GMS] J: installing view [J|0] (1) [J] 19:22:54,285 DEBUG (testng-Test:[]) [GMS] J: created cluster (first member). My view is [J|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl 19:22:54,322 DEBUG (testng-Test:[]) [Configurator] set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75 19:22:54,322 DEBUG (testng-Test:[]) [TCP_NIO2] thread pool min/max/keep-alive: 0/8/60000 use_fork_join=false, internal pool: 0/8/30000 (8 cores available) 19:22:54,349 DEBUG (testng-Test:[]) [GMS] address=K, cluster=ISPN, physical address=127.0.0.1:8601 19:22:54,350 TRACE (testng-Test:[]) [GMS] K: discovery took 0 ms, members: 1 rsps (1 coords) [done] 19:22:54,350 DEBUG (testng-Test:[]) [GMS] K: sending JOIN(K) to J 19:22:54,350 TRACE (testng-Test:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,350 TRACE (testng-Test:[]) [TCP_NIO2] 127.0.0.1:8601: connecting to 127.0.0.1:8600 19:22:54,369 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,380 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,390 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,400 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,410 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,421 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,431 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,468 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,503 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,503 TRACE (jgroups-8,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,503 TRACE (jgroups-10,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,503 TRACE (jgroups-12,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,503 TRACE (jgroups-14,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,504 TRACE (jgroups-16,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,504 TRACE (jgroups-18,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,506 TRACE (ViewHandler-6,J:[]) [GMS] J: joiners=[K], suspected=[], leaving=[], new view: [J|1] (2) [J, K] *** This is the first weird bit (bug?), J says it's mcasting the view but it doesn't send a message to K *** It still waits for 2 seconds for an ack, before sending the JOIN_RSP, though 19:22:54,506 TRACE (ViewHandler-6,J:[]) [GMS] J: mcasting view [J|1] (2) [J, K] (2 mbrs) 19:22:54,506 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to null, src=J, headers are GMS: GmsHeader[VIEW], NAKACK2: [MSG, seqno=1], TP: [cluster_name=ISPN] 19:22:54,506 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: looping back message [dst: , src: J (3 headers), size=57 bytes] 19:22:54,504 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,512 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received message batch of 1 messages from J 19:22:54,517 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,528 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,532 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=1, ts=1, TP: [cluster_name=ISPN] 19:22:54,538 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,547 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,547 TRACE (jgroups-20,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,547 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,547 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,549 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,549 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:54,550 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, ts=1, TP: [cluster_name=ISPN] 19:22:54,553 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=1, ts=2, TP: [cluster_name=ISPN] 19:22:54,556 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, ts=2, TP: [cluster_name=ISPN] 19:22:54,632 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to null, src=J, headers are NAKACK2: [HIGHEST_SEQNO, seqno=1], TP: [cluster_name=ISPN] 19:22:54,633 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: , src: J (2 headers), size=0 bytes, flags=OOB|INTERNAL], headers are NAKACK2: [HIGHEST_SEQNO, seqno=1], TP: [cluster_name=ISPN] 19:22:56,350 WARN (testng-Test:[]) [GMS] K: JOIN(K) sent to J timed out (after 2000 ms), on try 1 19:22:56,351 TRACE (testng-Test:[]) [GMS] K: discovery took 0 ms, members: 1 rsps (1 coords) [done] 19:22:56,351 DEBUG (testng-Test:[]) [GMS] K: sending JOIN(K) to J 19:22:56,353 TRACE (testng-Test:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,354 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=K, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,362 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=2, ts=3, TP: [cluster_name=ISPN] 19:22:56,362 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, ts=3, TP: [cluster_name=ISPN] 19:22:56,507 TRACE (ViewHandler-6,J:[]) [GMS] J: sending join-rsp to K: view=[J|1] (2) [J, K] (2 mbrs) 19:22:56,517 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,519 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (3 headers), size=61 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,519 DEBUG (testng-Test:[]) [GMS] K: installing view [J|1] (2) [J, K] 19:22:56,522 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are UNICAST3: ACK, seqno=1, ts=1, TP: [cluster_name=ISPN] 19:22:56,522 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, ts=1, TP: [cluster_name=ISPN] 19:22:56,540 TRACE (testng-Test:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:22:56,541 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are NAKACK2: [XMIT_REQ, sender=J], TP: [cluster_name=ISPN] 19:22:56,541 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:22:56,541 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=21 bytes, flags=OOB|INTERNAL], headers are NAKACK2: [XMIT_REQ, sender=J], TP: [cluster_name=ISPN] 19:22:56,543 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=3, ts=4, TP: [cluster_name=ISPN] 19:22:56,544 TRACE (testng-Test:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are RequestCorrelator: corr_id=200, type=REQ, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:22:56,544 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=7 bytes, flags=OOB|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=REQ, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:22:56,544 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=3, ts=4, TP: [cluster_name=ISPN] 19:22:56,544 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are GMS: GmsHeader[VIEW], NAKACK2: [XMIT_RSP, seqno=1], TP: [cluster_name=ISPN] 19:22:56,544 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received message batch of 1 messages from J 19:22:56,553 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=4, ts=5, TP: [cluster_name=ISPN] 19:22:56,553 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=4, ts=5, TP: [cluster_name=ISPN] 19:22:56,559 TRACE (ViewHandler-6,J:[]) [GMS] J: got all ACKs (1) from joiners for view [J|1] *** The second weird bit, J finally gets the ACK, and immediately creates a new view with the same members 19:22:56,562 TRACE (ViewHandler-6,J:[]) [GMS] J: joiners=[K], suspected=[], leaving=[], new view: [J|2] (2) [J, K] 19:22:56,562 TRACE (ViewHandler-6,J:[]) [GMS] J: mcasting view [J|2] (2) [J, K] (2 mbrs) 19:22:56,562 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to null, src=J, headers are GMS: GmsHeader[VIEW], NAKACK2: [MSG, seqno=2], TP: [cluster_name=ISPN] 19:22:56,562 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: looping back message [dst: , src: J (3 headers), size=60 bytes] 19:22:56,563 TRACE (remote-thread-Test-J-p35696-t2:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,563 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (3 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=RSP, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,562 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received message batch of 1 messages from J 19:22:56,562 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received message batch of 1 messages from J *** And the most weird, J and K receive the same message, but J sees a full view and K sees a delta view *** This smells like a concurrency issue to me 19:22:56,572 TRACE (jgroups-7,J:[]) [GMS] J: received full view: [J|1] (2) [J, K] 19:22:56,572 DEBUG (jgroups-7,J:[]) [GMS] J: installing view [J|1] (2) [J, K] 19:22:56,573 TRACE (jgroups-5,K:[]) [GMS] K: received delta view [J|2], ref-view=[J|0], joined=[K] 19:22:56,575 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are UNICAST3: ACK, seqno=2, ts=2, TP: [cluster_name=ISPN] 19:22:56,575 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, ts=2, TP: [cluster_name=ISPN] *** Shouldn't there be a way for the client GMS to signal to the coordinator that the delta view didn't work and it needs a full view? 19:22:56,573 WARN (jgroups-5,K:[]) [GMS] K: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([J|0]) doesn't match the current view-id ([J|1]); discarding delta view [J|2], ref-view=[J|0], joined=[K] 19:22:56,576 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,580 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: K, src: J (3 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=RSP, req_id=98983, rsp_expected=true, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:56,582 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: sending msg to J, src=J, headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=1, conn_id=1, first, TP: [cluster_name=ISPN] 19:22:56,582 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: looping back message [dst: J, src: J (3 headers), size=0 bytes, flags=OOB|INTERNAL] 19:22:56,582 TRACE (jgroups-7,J:[]) [GMS] J: received delta view [J|2], ref-view=[J|0], joined=[K] 19:22:56,582 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: J (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=1, conn_id=1, first, TP: [cluster_name=ISPN] 19:22:56,582 TRACE (ViewHandler-6,J:[]) [GMS] J: got all ACKs (1) from members for view [J|2] 19:22:56,583 TRACE (ViewHandler-6,J:[]) [GMS] J: sending join-rsp to K: view=[J|2] (2) [J, K] (2 mbrs) 19:22:56,583 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:22:56,583 TRACE (transport-thread-Test-J-p35698-t2:[ViewHandling]) [TCP_NIO2] J: sending msg to K, src=J, headers are RequestCorrelator: corr_id=200, type=REQ, req_id=98995, rsp_expected=true, UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:22:56,583 WARN (jgroups-7,J:[]) [GMS] J: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([J|0]) doesn't match the current view-id ([J|1]); discarding delta view [J|2], ref-view=[J|0], joined=[K] 19:22:56,585 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are UNICAST3: ACK, seqno=2, ts=3, TP: [cluster_name=ISPN] 19:22:56,585 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, ts=3, TP: [cluster_name=ISPN] 19:22:56,583 TRACE (jgroups-4,K:[]) [TCP_NIO2] K: received [dst: K, src: J (3 headers), size=7 bytes, flags=OOB|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=REQ, req_id=98995, rsp_expected=true, UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:22:56,583 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: K, src: J (3 headers), size=63 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:22:56,590 DEBUG (jgroups-5,K:[]) [GMS] K: installing view [J|2] (2) [J, K] 19:22:56,596 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are UNICAST3: ACK, seqno=4, ts=4, TP: [cluster_name=ISPN] 19:22:56,595 TRACE (remote-thread-Test-K-p35704-t2:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=98995, rsp_expected=true, UNICAST3: DATA, seqno=5, TP: [cluster_name=ISPN] 19:22:56,596 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=4, ts=4, TP: [cluster_name=ISPN] *** L joins 19:22:56,748 DEBUG (testng-Test:[]) [GMS] address=L, cluster=ISPN, physical address=127.0.0.1:8602 19:22:56,755 TRACE (testng-Test:[]) [GMS] L: discovery took 0 ms, members: 2 rsps (1 coords) [done] 19:22:56,776 DEBUG (testng-Test:[]) [GMS] L: sending JOIN(L) to J 19:22:56,787 TRACE (testng-Test:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,790 TRACE (testng-Test:[]) [TCP_NIO2] 127.0.0.1:8602: connecting to 127.0.0.1:8600 19:22:56,797 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,809 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,879 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,879 TRACE (jgroups-9,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,879 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,886 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are UNICAST3: ACK, seqno=1, ts=7, TP: [cluster_name=ISPN] 19:22:56,898 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,911 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,922 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,922 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, ts=7, TP: [cluster_name=ISPN] 19:22:56,943 TRACE (jgroups-4,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,943 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,943 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=1, first, TP: [cluster_name=ISPN] 19:22:56,950 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are UNICAST3: ACK, seqno=1, ts=8, TP: [cluster_name=ISPN] 19:22:56,955 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, ts=8, TP: [cluster_name=ISPN] 19:22:58,583 TRACE (ViewHandler-6,J:[]) [GMS] J: joiners=[L], suspected=[], leaving=[], new view: [J|3] (3) [J, K, L] 19:22:58,583 TRACE (ViewHandler-6,J:[]) [GMS] J: mcasting view [J|3] (3) [J, K, L] (3 mbrs) 19:22:58,583 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to null, src=J, headers are GMS: GmsHeader[VIEW], NAKACK2: [MSG, seqno=3], TP: [cluster_name=ISPN] 19:22:58,583 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: looping back message [dst: , src: J (3 headers), size=61 bytes] *** Now both J and K receive a delta view *** However, K still rejects it, even though the delta references view 1 and it was not able to install view 2 19:22:58,583 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received message batch of 1 messages from J 19:22:58,583 TRACE (jgroups-5,K:[]) [GMS] K: received delta view [J|3], ref-view=[J|1], joined=[L] 19:22:58,583 WARN (jgroups-5,K:[]) [GMS] K: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([J|1]) doesn't match the current view-id ([J|2]); discarding delta view [J|3], ref-view=[J|1], joined=[L] 19:22:58,583 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received message batch of 1 messages from J 19:22:58,585 TRACE (jgroups-5,J:[]) [GMS] J: received delta view [J|3], ref-view=[J|1], joined=[L] 19:22:58,585 DEBUG (jgroups-5,J:[]) [GMS] J: installing view [J|3] (3) [J, K, L] 19:22:58,583 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received message batch of 1 messages from J 19:22:58,596 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: sending msg to J, src=J, headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=2, conn_id=1, TP: [cluster_name=ISPN] 19:22:58,596 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: looping back message [dst: J, src: J (3 headers), size=0 bytes, flags=OOB|INTERNAL] 19:22:58,596 TRACE (transport-thread-Test-J-p35698-t3:[ViewHandling]) [TCP_NIO2] J: sending msg to null, src=J, headers are RequestCorrelator: corr_id=200, type=REQ, req_id=99326, rsp_expected=true, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ISPN] 19:22:58,596 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: J (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=2, conn_id=1, TP: [cluster_name=ISPN] 19:22:58,596 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: , src: J (3 headers), size=7 bytes, flags=OOB|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=REQ, req_id=99326, rsp_expected=true, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ISPN] 19:22:58,596 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: , src: J (3 headers), size=7 bytes, flags=OOB|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=REQ, req_id=99326, rsp_expected=true, NAKACK2: [MSG, seqno=4], TP: [cluster_name=ISPN] 19:22:58,598 TRACE (remote-thread-Test-K-p35704-t4:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99326, rsp_expected=true, UNICAST3: DATA, seqno=6, TP: [cluster_name=ISPN] 19:22:58,598 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: K (3 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99326, rsp_expected=true, UNICAST3: DATA, seqno=6, TP: [cluster_name=ISPN] 19:22:58,606 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to K, src=J, headers are UNICAST3: ACK, seqno=6, ts=9, TP: [cluster_name=ISPN] 19:22:58,606 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: K, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=6, ts=9, TP: [cluster_name=ISPN] 19:22:58,774 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to null, src=J, headers are NAKACK2: [HIGHEST_SEQNO, seqno=4], TP: [cluster_name=ISPN] 19:22:58,775 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: , src: J (2 headers), size=0 bytes, flags=OOB|INTERNAL], headers are NAKACK2: [HIGHEST_SEQNO, seqno=4], TP: [cluster_name=ISPN] 19:22:58,775 TRACE (jgroups-5,K:[]) [TCP_NIO2] K: received [dst: , src: J (2 headers), size=0 bytes, flags=OOB|INTERNAL], headers are NAKACK2: [HIGHEST_SEQNO, seqno=4], TP: [cluster_name=ISPN] 19:22:58,790 WARN (testng-Test:[]) [GMS] L: JOIN(L) sent to J timed out (after 2000 ms), on try 1 19:22:58,790 TRACE (testng-Test:[]) [GMS] L: discovery took 0 ms, members: 2 rsps (1 coords) [done] 19:22:58,791 DEBUG (testng-Test:[]) [GMS] L: sending JOIN(L) to J 19:22:58,791 TRACE (testng-Test:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:58,791 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_REQ]: mbr=L, UNICAST3: DATA, seqno=2, TP: [cluster_name=ISPN] 19:22:58,795 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are UNICAST3: ACK, seqno=2, ts=10, TP: [cluster_name=ISPN] 19:22:58,795 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, ts=10, TP: [cluster_name=ISPN] 19:22:59,877 TRACE (Timer runner-1,K:[]) [TCP_NIO2] K: sending msg to J, src=K, headers are STABLE: [STABLE_GOSSIP] view-id= [J|2], TP: [cluster_name=ISPN] 19:22:59,877 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: K (2 headers), size=41 bytes, flags=OOB|NO_RELIABILITY|INTERNAL], headers are STABLE: [STABLE_GOSSIP] view-id= [J|2], TP: [cluster_name=ISPN] 19:23:00,602 TRACE (ViewHandler-6,J:[]) [GMS] J: sending join-rsp to L: view=[J|3] (3) [J, K, L] (3 mbrs) 19:23:00,602 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=1, conn_id=2, first, TP: [cluster_name=ISPN] 19:23:00,603 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (3 headers), size=81 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=1, conn_id=2, first, TP: [cluster_name=ISPN] 19:23:00,603 DEBUG (testng-Test:[]) [GMS] L: installing view [J|3] (3) [J, K, L] 19:23:00,605 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are UNICAST3: ACK, seqno=1, conn_id=2, ts=1, TP: [cluster_name=ISPN] 19:23:00,605 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: L (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, conn_id=2, ts=1, TP: [cluster_name=ISPN] 19:23:00,610 TRACE (testng-Test:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:23:00,610 TRACE (jgroups-7,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=0 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[VIEW_ACK], UNICAST3: DATA, seqno=3, TP: [cluster_name=ISPN] 19:23:00,611 TRACE (ViewHandler-6,J:[]) [GMS] J: got all ACKs (1) from joiners for view [J|3] 19:23:00,611 TRACE (ViewHandler-6,J:[]) [GMS] J: L already present; returning existing view [J|3] (3) [J, K, L] 19:23:00,611 TRACE (ViewHandler-6,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=2, conn_id=2, TP: [cluster_name=ISPN] 19:23:00,611 TRACE (testng-Test:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are RequestCorrelator: corr_id=200, type=REQ, req_id=99746, rsp_expected=true, UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:23:00,611 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=7 bytes, flags=OOB|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=REQ, req_id=99746, rsp_expected=true, UNICAST3: DATA, seqno=4, TP: [cluster_name=ISPN] 19:23:00,611 TRACE (ViewHandler-6,J:[]) [GMS] J: found no members to add or remove, will not create new view 19:23:00,611 TRACE (jgroups-5,L:[]) [TCP_NIO2] L: received [dst: L, src: J (3 headers), size=81 bytes, flags=OOB|INTERNAL], headers are GMS: GmsHeader[JOIN_RSP], UNICAST3: DATA, seqno=2, conn_id=2, TP: [cluster_name=ISPN] 19:23:00,612 TRACE (remote-thread-Test-J-p35696-t5:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99746, rsp_expected=true, UNICAST3: DATA, seqno=3, conn_id=2, TP: [cluster_name=ISPN] 19:23:00,612 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (3 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99746, rsp_expected=true, UNICAST3: DATA, seqno=3, conn_id=2, TP: [cluster_name=ISPN] 19:23:00,614 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are UNICAST3: ACK, seqno=4, ts=11, TP: [cluster_name=ISPN] 19:23:00,615 TRACE (Timer runner-1,L:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are UNICAST3: ACK, seqno=3, conn_id=2, ts=2, TP: [cluster_name=ISPN] 19:23:00,616 TRACE (remote-thread-Test-L-p36073-t2:[]) [TCP_NIO2] L: sending msg to J, src=L, headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99326, rsp_expected=true, UNICAST3: DATA, seqno=5, TP: [cluster_name=ISPN] 19:23:00,616 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: L (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=3, conn_id=2, ts=2, TP: [cluster_name=ISPN] 19:23:00,616 TRACE (jgroups-5,J:[]) [TCP_NIO2] J: received [dst: J, src: L (3 headers), size=5 bytes, flags=OOB|NO_FC|NO_TOTAL_ORDER], headers are RequestCorrelator: corr_id=200, type=RSP, req_id=99326, rsp_expected=true, UNICAST3: DATA, seqno=5, TP: [cluster_name=ISPN] 19:23:00,618 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=4, ts=11, TP: [cluster_name=ISPN] 19:23:00,625 TRACE (Timer runner-1,J:[]) [TCP_NIO2] J: sending msg to L, src=J, headers are UNICAST3: ACK, seqno=5, ts=12, TP: [cluster_name=ISPN] 19:23:00,625 TRACE (jgroups-4,L:[]) [TCP_NIO2] L: received [dst: L, src: J (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=5, ts=12, TP: [cluster_name=ISPN]