There are currently two designs for this:
1. Use JGroups' RELAY for master/master.
2. Use an Infinispan-only setup using Hot Rod and an async RemoteCacheStore
Both design patterns should be documented, tested, benchmarked. Should survive failures in relaying node, on either datacentre.
Initially test on 2 datacentres, and if possible on 3.