[1] was about removing UNICAST3 / NAKACK2, along other protocols such as MFC / UFC etc, when running over TCP or TCP_NIO2.

When I started implementing this, I ran into multiple issues. The most worrying thing is that there is a long tail of smaller issues that need to be identified and tackled, in order not to (a) lose a single message and (b) preserve sender-order.

This led me to believe that it is not feasible to run without UNICAST-X and NAKACK-X. The amount of work to close all loopholes would have been too much.

I'll therefore close the remaining open issues and revert code added by [1] and its fanout issues.

Issues

• Any protocol (also user-written ones) or component can drop/reorder messages. These would all have to be identified and fixed.
•  TCP and TCP_NIO2 send multicast messages (e.g. to B, C) in turn: first to B then to C. If the send to B fails, we'd have to do repair, before sending to C. Otherwise, this could lead to message reordering
• In TCP_NIO2, we don't block if the send to B is partial, but go on and send to C. The selector will wake us up when the write to B is available (ie., space). This also leads to reordering
• Protocols such as RED drop messages: they would not be usable. Also, user-written protocols would have to make sure to not reorder, drop or send duplicate messages
• The thread pool cannot drop messages: we'd have to use a CallerRuns rejection policy
• The MessageProcessingPolicy cannot drop messages: however, MaxOneThreadPerSender does drop messages if the queue is bounded and full. [2] was created for this.
• A Bundler which drops messages, cannot be used. TransferQueueBundler would have to have a lock around sending multicast messages over TCP, as described above.
• Without digests (provided by NAKACK-X), state transfer is not possible.
• Merging is not possible with NAKACK-X absent!
   

[1] https://issues.redhat.com/browse/JGRP-2566
[2] https://issues.redhat.com/browse/JGRP-2686