ClusterStreamManager has to find all of the addresses for a given set of segments. Unfortunately it does this by finding the segments per address. This ends up having n * m time complexity. If we instead did it so we find the primary owner per segment it would only have n time complexity as finding the primary owner of a segment is constant time.