Uploaded image for project: 'ModeShape'
  1. ModeShape
  2. MODE-2486

Investigate Lucene index memory usage when indexing nodes that have binary values, text extraction is disabled and full text search is enabled

    Details

      Description

      Lucene memory usage should be investigated when indexing nodes that have binary properties full text search is enabled and text extraction is not enabled. In this particular case, there is no reason for ModeShape to attempt reading any binary content since the only meaningful (and required by the JCR spec) information is the length of the property.

      If text extraction is enabled, then the binary content will have to be read by the text extractor(s) and there is no way around this

      If OOM exceptions in the form of:

      0:00:43,906 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Hibernate Search sync consumer thread for index nodeinfo) HSEARCH000058: Exception occurred java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
      Primary Failure:
      	Entity org.modeshape.jcr.query.lucene.basic.NodeInfo  Id d7d2e6b7505d64aabe99db-872d-4a93-b68d-8ae175a3a971  Work Type  org.hibernate.search.backend.AddLuceneWork
      : java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit
      	at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3358) [lucene-core-3.6.2.redhat-8.jar:3.6.2.redhat-8]
      	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3485) [lucene-core-3.6.2.redhat-8.jar:3.6.2.redhat-8]
      	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.redhat-8.jar:3.6.2.redhat-8]
      	at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.redhat-8.jar:3.6.2.redhat-8]
      	at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:158) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:171) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.PerChangeSetCommitPolicy.onChangeSetApplied(PerChangeSetCommitPolicy.java:46) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.AbstractWorkspaceImpl.afterTransactionApplied(AbstractWorkspaceImpl.java:107) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:124) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.applyChangesets(SyncWorkProcessor.java:162) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at org.hibernate.search.backend.impl.lucene.SyncWorkProcessor$Consumer.run(SyncWorkProcessor.java:152) [hibernate-search-engine-4.6.0.Final-redhat-2.jar:4.6.0.Final-redhat-2]
      	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
      

      occur, there are a couple of workarounds that can be used:
      1. make the node which holds the jcr:data property not queryable. For example:

      [nt:nqresource] > nt:resource noquery
      

      2. disable full search search from the JSON configuration:

       "query" : {
              "enabled" : true,
              "enableFullTextSearch" : false,
              }
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                hchiorean Horia Chiorean
                Reporter:
                hchiorean Horia Chiorean
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: