Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-11512

The mass indexer should iterate over the cache only once

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Done
    • Major
    • 11.0.0.Final, 10.1.6.Final
    • 10.1.5.Final, 11.0.0.Dev03
    • Indexing
    • None

    Description

      DistributedExecutorMassIndexer starts an IndexWorker for each indexed type and submits them to all the nodes in parallel. Each IndexWorker runs a blocking iteration over the cache, and when the cache has a store, that iteration is very expensive.

      Each iteration will load and deserialize all the entries in the store. Most stores don't implement AbstractSegmentedStoreConfiguration, so Infinispan wraps them in a ComposedSegmentedLoadWriteStore, which iterates over each segment in parallel on the persistence/blocking executor. Since the default number of segments is 256 and the persistence/blocking executor has 4*cpu_count/150 max threads, it doesn't take a lot of parallel iterations to fill the blocking executor's threads and prevent the cache from doing other, more urgent, work.

      Attachments

        Issue Links

          Activity

            People

              gfernand@redhat.com Gustavo Fernandes (Inactive)
              dberinde@redhat.com Dan Berindei (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: