Loading...

Details

Type: Bug
Resolution: Done
Priority: Minor
Fix Version/s: 5.1
Affects Version/s: 5.0.0.Final
Labels:
None

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Steps to Reproduce:
Hide

I included a test program based on the SimpleChat jgroups example. (I am not a Java developer, so please excuse any idiosyncrasy of the code...)

Create two physically distant Linux servers. I used two newly built CentOS 8 Linodes, one in Fremont, CA, and the other in Newark, NJ. Ping time between the servers is ~65 milliseconds.

Configure net.core.rmem_max and net.core_wmem_max to something large such as 32MB:

sudo sysctl -w net.core.rmem_max=33554432 net.core.wmem_max=33554432

Copy the following files to both servers:

SpeedTest.class

jgroups-5.0.0.Final.jar

tcp-5.0.0.xml (a copy of tcp.xml in the jgroups-5.0.0.Final.jar)

Configure send_buf_size and recv_buf_size in tcp-5.0.0.xml:

<TCP... send_buf_size="33554432" recv_buf_size="33554432"/>

Run SpeedTest on both machines and wait for them to connect:

java -Djgroups.tcpping.initial_hosts=jgroups-west[7800],jgroups-east[7800] -cp jgroups-5.0.0.Final.jar:. SpeedTest

On either machine, enter the command "send" or "recv" to have that machine receive (or send) 16MB and output the estimated throughput in bytes/sec. One of these will be significantly slower than the other, and will correspond to data sent from client (connect side) to server (listen side)

Example:

[jgroups@jgroups-west ~]$ java -Djgroups.tcpping.initial_hosts=jgroups-west[7800],jgroups-east[7800] -cp jgroups-5.0.0.Final.jar:. SpeedTest Sep 29, 2020 6:16:59 PM org.jgroups.JChannel setAddress INFO: local_addr: d1845247-6de6-d80a-14cd-78524a0925fe, name: jgroups-west-24449 ------------------------------------------------------------------- GMS: address=jgroups-west-24449, cluster=SpeedTestCluster, physical address=45.79.68.10:7800 ------------------------------------------------------------------- ** view: [jgroups-east-34095|1] (2) [jgroups-east-34095, jgroups-west-24449] === NOTE - This instance is currently the connect() side === > send Sending... > Sent 16777216 bytes at 2699498 bytes/sec recv Receiving... > Received 16777216 bytes at 15127927 bytes/sec === NOTE - Stopped and restarted the remote side === > ** view: [jgroups-west-24449|2] (1) [jgroups-west-24449] ** view: [jgroups-west-24449|3] (2) [jgroups-west-24449, jgroups-east-47558] === NOTE - This instance is now the listen() side === > send Sending... > Sent 16777216 bytes at 14863557 bytes/sec recv Receiving... > Received 16777216 bytes at 2626508 bytes/sec
Show
I included a test program based on the SimpleChat jgroups example. (I am not a Java developer, so please excuse any idiosyncrasy of the code...) Create two physically distant Linux servers. I used two newly built CentOS 8 Linodes, one in Fremont, CA, and the other in Newark, NJ. Ping time between the servers is ~65 milliseconds. Configure net.core.rmem_max and net.core_wmem_max to something large such as 32MB: sudo sysctl -w net.core.rmem_max=33554432 net.core.wmem_max=33554432 Copy the following files to both servers: SpeedTest.class jgroups-5.0.0.Final.jar tcp-5.0.0.xml (a copy of tcp.xml in the jgroups-5.0.0.Final.jar) Configure send_buf_size and recv_buf_size in tcp-5.0.0.xml: <TCP... send_buf_size= "33554432" recv_buf_size= "33554432" /> Run SpeedTest on both machines and wait for them to connect: java -Djgroups.tcpping.initial_hosts=jgroups-west[7800],jgroups-east[7800] -cp jgroups-5.0.0.Final.jar:. SpeedTest On either machine, enter the command "send" or "recv" to have that machine receive (or send) 16MB and output the estimated throughput in bytes/sec. One of these will be significantly slower than the other, and will correspond to data sent from client (connect side) to server (listen side) Example: [jgroups@jgroups-west ~]$ java -Djgroups.tcpping.initial_hosts=jgroups-west[7800],jgroups-east[7800] -cp jgroups-5.0.0.Final.jar:. SpeedTest Sep 29, 2020 6:16:59 PM org.jgroups.JChannel setAddress INFO: local_addr: d1845247-6de6-d80a-14cd-78524a0925fe, name: jgroups-west-24449 ------------------------------------------------------------------- GMS: address=jgroups-west-24449, cluster=SpeedTestCluster, physical address=45.79.68.10:7800 ------------------------------------------------------------------- ** view: [jgroups-east-34095|1] (2) [jgroups-east-34095, jgroups-west-24449] === NOTE - This instance is currently the connect() side === > send Sending... > Sent 16777216 bytes at 2699498 bytes/sec recv Receiving... > Received 16777216 bytes at 15127927 bytes/sec === NOTE - Stopped and restarted the remote side === > ** view: [jgroups-west-24449|2] (1) [jgroups-west-24449] ** view: [jgroups-west-24449|3] (2) [jgroups-west-24449, jgroups-east-47558] === NOTE - This instance is now the listen() side === > send Sending... > Sent 16777216 bytes at 14863557 bytes/sec recv Receiving... > Received 16777216 bytes at 2626508 bytes/sec

SFDC Cases Counter:
SFDC Cases Links:

Description

I recently finished troubleshooting a unidirectional throughput bottleneck involving a JGroups application (Infinispan) communicating over a high-latency (~45 milliseconds) TCP connection.

The root cause was JGroups improperly configuring the receive/send buffers on the listening socket. According to the tcp(7) man page:

On individual connections, the socket buffer size must be set prior to
the listen(2) or connect(2) calls in order to have it take effect.

However, JGroups does not set the buffer size on the listening side until after accept().

The result is poor throughput when sending data from client (connecting side) to server (listening side.) Because the issue is a too-small TCP receive window, throughput is ultimately latency-bound.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

bla5.java
1 kB
2020/09/30 6:54 AM
bla6.java
2 kB
2020/09/30 7:21 AM
bla7.java
2 kB
2020/09/30 7:34 AM
delay-ip.sh
0.6 kB
2020/09/30 1:38 PM
rcvbuf.png
72 kB
2020/10/06 12:40 PM
SpeedTest.java
4 kB
2020/09/29 2:25 PM

Poor throughput over high latency TCP connection when recv_buf_size is configured

Details

Description

Attachments

Attachments

Activity

People

Dates