JGroups routes messages with the assumption of a flat network, e.g. in {A,B,C,D}, a message sent by A to all is sent to B, then to C, then to D (N-1 approach in TCP).

This is fine if the cost of sending a message is the same for all members.

However, if we had a topology where the cost wasn't the same, flat routing would be inefficient. Example:

• Hosts A, B, C
• Members A1, A2, A3 on A, B1,B2,B3 on B and C1,C2,C3 on C
• With a shared memory transport (JGRP-1672) in place, the local processes on a host (e.g. B1,B2,B3) would use shared memory to communicate, which is faster than TCP/IP for local processes on the same host. TCP would only be used to communicate between members on different hosts, e.g. A2 talking to B1.

Currently, B2 sending a message to all group members would send it to B1, B3, A1-3 and C1-3 (N-1 approach).

With topology information available, B2 could send the message only to a given member (fixed or dynamically chosen) of each host, e.g. to A1 and C3, and then deliver the message locally (via shared memory) to B1 and B3. A1 would delivery the message locally to A2/A3, and C3 to C1/C2.

This reduces the number of trips across hosts from 6 to 2, and should therefore be faster.

The idea would be to have an abstract routing protocol ROUTE (above the transport) which calls (abstract) methods nextHop() -> Address when sending a message, copies/wraps the message and forwards it to the next hop(s), unwraps it and calls deliverForwardedMessage/Batch() when the message has been received.

A subclass of ROUTE could then implement shared memory based routing, as described above.

Another subclass might implement daisy chaining (https://github.com/belaban/JGroups/blob/master/doc/design/CLOUD_TCP.txt).

We might even consider cross-site replication (RELAY2), where we have one big cluster across sites rather than two separate clusters, but where messages are routed locally (within a site) as much as possible to reduce cross-site trips.