When a node stores its information in a directory (FILE_PING, S3_PING or GOOGLE_PING), then we can optimize discovery by implementing a few things:

• After reading all files, we send each node (represented by a file) a discovery request. That node processes the request and sends back a discovery response. This is unneeded traffic, especially with large clusters. Instead
  • Read all files and add the information read from the files into the local caches (logical_addr_cache, UUID cache etc). This is the same as processing discovery responses from all members
• Determine the coordinators directly from the file information. Perhaps we could even create a special file which contains information about the coordinator.
  • This would prevent partitions from happening when starting up a large number of nodes: as long as that special file exists, nobody else will take ownership of it. When the coord leaves or crashes, we atomically replace the special file