-
Bug
-
Resolution: Done
-
Major
-
3.6.9
-
None
11:24:19,178 TRACE (OOB-10,NodeD-30623:[]) [MFC] NodeA-6644 used 4969 credits, 396763 remaining 11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [MFC] sending 603237 credits to NodeA-6644 11:24:19,238 TRACE (OOB-10,NodeD-30623:[]) [UDP] NodeD-30623: sending msg to NodeA-6644, src=NodeD-30623, headers are MFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default] 11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UDP] NodeA-6644: received [dst: NodeA-6644, src: NodeD-30623 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are UFC: REPLENISH, UNICAST3: DATA, seqno=8825, TP: [cluster_name=default] 11:24:19,246 TRACE (INT-1,NodeA-6644:[]) [UFC] received 603237 credits from NodeD-30623, old credits: 938772, new credits: 1000000
15:20:14,596 TRACE (OOB-44,NodeC-64981:[]) [UFC] NodeA-12977 used 2305 credits, 798175 remaining 15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UFC] sending 1201825 credits to NodeA-12977 15:20:14,614 TRACE (OOB-44,NodeC-64981:[]) [UDP] NodeC-64981: sending msg to NodeA-12977, src=NodeC-64981, headers are UFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default] 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UDP] NodeA-12977: received [dst: NodeA-12977, src: NodeC-64981 (3 headers), size=8 bytes, flags=OOB|DONT_BUNDLE|INTERNAL], headers are MFC: REPLENISH, UNICAST3: DATA, seqno=34854, TP: [cluster_name=default] 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [UFC] NodeC-64981 used 8 credits, 1859662 remaining 15:20:14,629 TRACE (INT-1,NodeA-12977:[]) [MFC] received 1201825 credits from NodeC-64981, new credits for NodeC-64981: 2000000, min_credits=793978
UFC/MFC expect their messages to be reliable, so they don't repeat REPLENISH messages, and the only way to replenish the credits is to send a CREDIT_REQUEST message. But UFC/MFC only send a credit request after all the threads trying to send a request or response to the affected destination block for max_block_time.
I was able to reproduce the problem with an Infinispan micro-benchmark: a cluster of 4 nodes, a replicated cache, 400 threads writing to the cache. It happens both with JGroups 3.6.9.Final and the latest master.
