Based on the discussion below, consider all topology cache updates to be done by coordinator to avoid concurrency issues when updating it.

> Looks good.
> What is causing this unsuccessful add? If it is caused by timeouts due
> to multiple caches operating on the same key an alternative would be
> to only perform the operation on the coordinator and rest of the
> members to have node added listeners ...

Currently, each node when it starts, it's responsible of adding itself to the view and when it stops, it's responsible from removing itself. Apart from this, there's a crashed member listener running only in coordinator that detects whether any member left without updating the topology view. Your suggestion to have the coordinator control it all seems like could work and get around potential timeouts.

I'll create a JIRA to investigate this but won't do it for CR1 since I'm expect this to be a major issue. Metadata size is small and it's not constantly uddated.