A cache with await-initial-transfer="false" will execute cache operations while it is receiving state, during rebalance phase READ_OLD_WRITE_ALL. State transfer can take a long time, and during that time, it is not a read owner for any segments, it is only a write owner for the segments it is receiving.

That means cache.get(k) will perform a remote lookup every time, even AFTER the node received entry k=v via transfer, as long as not all the nodes have confirmed the end of state transfer and the coordinator hasn't changed the rebalance phase to READ_ALL_WRITE_ALL.

The extra remote lookups can have a negative impact on application performance. Especially in a replicated cache, the application would expect reads to be very fast, and the repeated remote lookups would break that assumption.

In order to encourage await-initial-transfer="false" and eventually make it the default (ISPN-9112), we should limit the number of remote lookups performed by a node while is is a write-only owner of a key:

• Remote reads should write the entry in the data container, the same way state transfer would (i.e. skipping the write if a write operation already changed the entry).
• Reads should first look up the key in the local data container before going remotely. If the key exists locally, the value can be returned directly.