Adding a clear and iteration fork to the SoftIndexFileStoreStressTest and removing the remove and compaction forks allowed me to find an issue with clear being ran concurrently with other operations.

The exception found is:

14:45:18,689 ERROR (testng-SoftIndexFileStoreStressTest:[]) [o.i.c.t.TestSuiteProgress] Test failed: org.infinispan.persistence.sifs.SoftIndexFileStoreStressTest.testConstantReadsWithCompaction

java.util.concurrent.ExecutionException: org.infinispan.commons.CacheException: Concurrent update to compacting file when reading metadata on 65220: 0: [keyLength=12, valueLength=278, metadataLength=0, internalMetadataLength=0,seqId=82862, expiration=-1]|0

The issue in this case was that the index clear step acquires the write lock to update the indexes but releases it too early, allowing other operations to be ran concurrently. Iteration never acquires the parent lock either. In this case the iteration step sees the index before clear but then when it attempts to read the data it is no longer present.

We need to change index clear to hold the lock until the CompletionStage completes. And the iteration portion either needs to acquire the lock for the entire operation or per segment. The latter would be better for performance but could starve a clear/stop if iterations keep being sent.