Major Description
When many threads are sending messages via TCP, and a destination is stuck (e.g. like in [1]), then a single thread (which acquired the lock) is blocked on the TCP write because the send-window is 0. All other senders are blocked trying to acquire the lock.
While TCP cannot be made non-blocking, we can at least change the above behavior to one in which only a single thread is blocked. This can be done in 2 ways:
Bounded queue with single consumer
Here, every TcpConnection has a bounded queue and a consumer thread which continually dequeues messages and sends them. Sender threads simply add their messages to the queue, or drop it if the queue is full. If the consumer thread is blocked on a full TCP send-window, then all others sender threads will simply drop their messages. This is not an issue, as retransmission makes sure that all dropped messages are retransmitted.
The benefit here is that - if we for example send messages one-by-one to A,B,C and D (B is blocked), then we would not be blocked trying to send to B and therefore also not send to (unblocked) C and D, but instead simply skip B and send to C and D.
Use of tryLock()
Instead of calling Lock.lock() before sending data, senders would use tryLock(timeout). When the lock cannot be acquired, the sender would simply return and thus drop the message. As with the avove solution, dropped messages will get retransmitted.
Initially, the timeout would be the default value. For each failed lock acquitision, the timeout would be reduced (halved?), so that after some time, a blocked sender thread would cause all other senders to drop the message almost immediately.
A successful lock acquisition would increase the timeout value again, so that no messages are dropped.
Advantage to the above algorithm: no queues and not the one consumer thread / TcpConnection. Disadvantage: how to compute the timeout correctly, might be complex. Plus, threads are cheap when using virtual threads.
