2) #1 alone will hurt performance in a steady-state system. If A and B are sending messages to each other, with a config of max_credits = 1,000,000 and min_credits=100,000, B will send A 900,000 credits when it has read that many. It will take a while for the credit replenishment to reach A (since B is also sending), so A will send 100,000 more and begin blocking. It will then get the 900,000 credit replenishment, send 900,000 and begin blocking again. Under the old system it would have gotten 1,000,000 bytes – now it only gets 900,000.

Solution to that is to change the meaning of min_credits. Currently, receiver sends credit when it has received_bytes >= (max_credits - min_credits). If min_credits is 100,000, credits will only be sent when the sender is almost out. The standard value of min_threshold=0.10 is very conservative, but was needed because the more frequently replenishment messages with no set number of bytes get sent, the more likely the sender is to get too many credits and OOME.

If credit replenishment messages only give the number of bytes the receiver has read, then there is no OOME risk. Therefore, sending replenishment messages frequently makes sense. So, I propose the receiver should send credit when received_bytes >= min_credits, rather than the current approach of sending when received_bytes >= max_credit - min_credit.