When a rebalance removes a segment X from node A, node A keeps updating entries in segment X until the rebalance finishes, and only deletes the entries of segment X after entering the NO_REBALANCE phase.
This is problematic for tests that work with the data container directly, because waitForNoRebalance() doesn't wait for the removal of stale entries. The test will work without an explicit wait most of the time, so this is a recipe for random test failures (e.g.
As described in
ISPN-5021, we can prevent any writes to segment X at the start of the READ_NEW_WRITE_ALL phase, send the phase confirmation to the coordinator, and then remove the entries asynchronously. We just need to keep track of the removal task and only install/confirm the NO_REBALANCE phase once all the entries that we don't own have been removed.