Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-4650

MassIndexer should not use UpdateDocument when adding to Lucene

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 7.1.0.Final
    • 7.0.0.Beta1
    • Embedded Querying
    • None

      The MassIndexer currently causes a Delete plus and Add operation to hibernate search backend.
      Lucene buffers those deletes queries and during merge it tries to 'apply' those deletes wasting a massive amount of time doing seeks and queries unnecessarily.
      Since the mass indexer wipes the index at the beginning, it should simply issue an add operation (or at least rely on Lucene atomic IndexWriter.updateDocument). Performance wise this make a huge difference:

      • indexing 50k documents brings down the indexing time from 195s to 33s
      • indexing 200k documents brings down the indexing time from 600s to 55s

              gfernand@redhat.com Gustavo Fernandes (Inactive)
              gfernand@redhat.com Gustavo Fernandes (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: