-
Enhancement
-
Resolution: Unresolved
-
Major
-
None
-
None
Today when the SIFS writes a value it does the following
1. Write to the append only log
2. Update TemporaryTable for index to allow return
3. Update index file, sequentially
4. Remove from TemporaryTable
To reduce performance costs 2-4 are separated into "Segments" to allow for parallel updates to the index and their respective files.
Instead we should be able to change #3 to do a batch update and not do them sequentially. Instead of submitting an UpdateIndex request for every operation, we could instead send a generic event to the Indexer to wake up. When this happens the Indexer would read the contents of the TemproraryTable and persist all updates to the index in one go. Then the index is written to disk once for a single batch, instead of once per write. This should improve scalability considerably for indexing.