OK, so setting SO_RCVBUF works now, good to know. JGroups also sets SO_SNDBUF on sockets, but always before calling connect (client side). Also, setting SO_SNDBUF on a socket received as result of calling accept() apparently works. Besides, as discussed, there is no way of setting this on a ServerSocket in Java...
Thanks for your detailed analysis; always great to work with experts in the field!
Note to self: I should make myself familiar with the Linux networking code, all the more since I have kernel guys in my company that I can ask for advice!
Cheers,

net.core.rmem_max is the limit on how large of a SO_RCVBUF an application may request.  If the application attempts to request a larger buffer, it is clamped to the value specified in net.core.rmem_max.

Buffer size and window size are related but distinct.  The buffer size is how much kernel memory is allocated to the socket; the receive window (a number advertised in the TCP header, indicating how much more data it is willing to receive at a given moment in time) depends on the buffer size and other factors.

Setting SO_SNDBUF on a listening socket works without issue in C.  Here is strace output of the "iperf3" utility setting up its listening socket with a requested buffer size of 123456.  The setsockopt calls all return 0 (success):

socket(AF_INET6, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [123456], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [123456], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(3, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
bind(3, {sa_family=AF_INET6, sin6_port=htons(7800), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
listen(3, 5)                            = 0

But based on what I've been able to piece together, it's completely OK – at least with Linux – to wait until after accepting a connection to configure SO_SNDBUF.  Considering that Java's ServerSocket lacks a setSendBufferSize method, I think it's safe to assume that it's OK on other platforms as well.

OK, so this means setting SO_RCVBUF replaces net.core.rmem_max? And the initial buffer size is the default size set in sysctl.conf?

Speaking of buffers: I assume buffer size == TCP recv/send window size?

Last point: setting SO_SNDBUF on a server socket in Java is not supported:

srv_sock.setOption(StandardSocketOptions.SO_RCVBUF, receive_buf_size) throws an exception. I haven't been in C land for quite a while, but do you know if this is supported in C?
Cheers,

I tested TCP and TCP_NIO2, and the throughput issue is now resolved.  Thank you for looking into this!

OK, so I read your comment. I understand socket options are inherited, but what sense does SO_SNDBUF make here? This option cannot be set on a ServerSocket anyway (perhaps this can be done in C?), only SO_RCVBUF, so does this mean the send-buffer cannot be set on a socket returned by accept()?

That part of the documentation didn't make sense to me either.  Reading through the Linux sources and verifying empirically, the SO_SNDBUF option is definitely being honored, even when it is configured on the accepted socket.

Also:

Does this mean, the value of SO_RCVBUF is the max value a receive window can have?

Yes.  SO_RCVBUF controls the buffer size, which in turn limits the receive window.  (The receive window grows/shrinks over time in response to quantity of data received, congestion and packet loss, the rate at which the receiving application consumes the data, etc.)  To illustrate, I configured both sites with SO_RCVBUF=200000 and graphed the window size over time during a send/receive test.

More Info: http://diag.ddns.net/reports/TCPWindows.html

OK, so I read your comment. I understand socket options are inherited, but what sense does SO_SNDBUF make here? This option cannot be set on a ServerSocket anyway (perhaps this can be done in C?), only SO_RCVBUF, so does this mean the send-buffer cannot be set on a socket returned by accept()?

Also:

If the application explicitly configures SO_RCVBUF, this automatic management is disabled

Does this mean, the value of SO_RCVBUF is the max value a receive window can have?

I also changed the Jira URL, this will show up in the next update of the web site.

OK, it should work now, try it out and let me know.

Cheers,

Taking me a little longer; I'm taking the opportunity for some refactoring.

I'll reply in detail later, but I know what the problem is: I thought I was using TCP.recv_buf_size (which is set correctly), but instead I used TCP.TcpServer.recv_buf_size which is set after bind(). I'll fix this today.