TP.logical_addr_cache maintains mappings between UUIDs and IP addresses. When an address is removed from it, it is not really removed but only marked as 'removable'.
When the cache exceeds its capacity, all removable values are removed.
However, we also have a cache reaper, which periodically (every logical_addr_cache_expiration ms) removes all values marked as removable. The default is 2 minutes. This means that the reaper runs every 2 minutes and drops removable entries that are older than 2 minutes.
If we have an entry for member P which was added 2 hours ago, and now member P leaves, then that entry is marked as removable. If the reaper happens to kick in just after P left, P's entry will be removed immediately !
SOLUTION:
- Update the timestamp of an entry when it is marked from normal to removable (don't mark it twice !)
- Allow for 0 as a valid reaper interval (disabling reaping)
- Introduce a reaper_interval (when the reaper runs) and logical_addr_cache_expiration (already exists)
- The former has to be smaller than the latter