When a cache is configured with SIFS in a DIST cache it is supposed to remove old data when a segment is no longer owned by the cache. Unfortunately, SIFS does not do that and thus data lingers in the index and data files even after those segments are removed. This means that if a segment is readded later that data will be resurrected as well causing stale reads as well as possibly preventing compaction.

The current code base would need to iterate over all of the segments that were removed and invoke synthetic SIFS remove operations to update the data files and the indexes appropriately.
If https://issues.redhat.com/browse/ISPN-15377 is instead used we can do this by instead listening for concurrent index updates and update the compactor size as well as iterating over the index file sequentially skipping unused blocks to update compactor free sizes. This should be substantially faster as we don't have to create a synthetic remove for every entry in the segment.