High Availability for non-shared indexes on DIST caches

'Non-shared indexes' is a setup when each node has a local index backed by the file system, using either the near-real-time or the FS index managers.

For REPL caches, a full copy of the index is kept in each node. As long as at least on node is up the queries hit the local index which represents the full index of the data.

But non-shared indexes have some drawbacks for DIST caches: each node only indexes data that the node is the primary owner. If a node goes down, there is no redundancy. This issue aims to solve this issue at two levels:

• Indexing: replicated entries should be indexed for redundancy purposes. If a primary owner goes down, the entry is indexed in the backup owner. This should follow exactly the data distribution on the cache in terms of number of replicas and rebalancing
• Querying: query broadcast is currently used for non-shared indexes. If replicated entries start to be indexed, a mechanism will have to be used to avoid retuning repeated results, since currently the broadcast simply queries all the entries in each node and join the result.