Investigate whether an RDMA based transport makes sense.

Advantages:

• Speed, low latency (TCP/IP is bypassed entirely)
• Low CPU usage

Disadvantages:

• JNI/C code
  • Such a transport implementation would have to live outside of the JGroups repo
  • Maintainability nightmare: the C code would also have to be ported to various OSes
    • Investigate Java based libs (IBM's jVerbs) and C based libs (Apache Portable Runtime?)
• High memory use, growing with cluster size: similarly to TCP, a 'group multicast' would involve N-1 sends. RDMA requires a Queue Pair (QP) for each destination. Each QP requires pinned memory (receive and send buffer), so each node would have to reserve (pin) N-1 memory buffers [1]
  • OTOH, we may not use many group multicasts, e.g. with Infinispan's partial replication (DIST mode)
• High cost of RDMA adapters, NICs and wiring: only a very small fraction of users would run such a transport.

[1] http://www.hpcwire.com/hpcwire/2006-08-18/a_critique_of_rdma-1.html