Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-11512

The mass indexer should iterate over the cache only once

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 11.0.0.Final, 10.1.6.Final
    • 10.1.5.Final, 11.0.0.Dev03
    • Indexing
    • None

      DistributedExecutorMassIndexer starts an IndexWorker for each indexed type and submits them to all the nodes in parallel. Each IndexWorker runs a blocking iteration over the cache, and when the cache has a store, that iteration is very expensive.

      Each iteration will load and deserialize all the entries in the store. Most stores don't implement AbstractSegmentedStoreConfiguration, so Infinispan wraps them in a ComposedSegmentedLoadWriteStore, which iterates over each segment in parallel on the persistence/blocking executor. Since the default number of segments is 256 and the persistence/blocking executor has 4*cpu_count/150 max threads, it doesn't take a lot of parallel iterations to fill the blocking executor's threads and prevent the cache from doing other, more urgent, work.

            gfernand@redhat.com Gustavo Fernandes (Inactive)
            dberinde@redhat.com Dan Berindei (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: