Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-1002

"org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JDG 7.0.1
    • JDG 7.0.0 GA, JDG 7.1.0 GA
    • Querying
    • None
    • DR2
    • Hide

      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven
      2. Deploy it to a standalone EAP 6.4 server.
      3. Run the following requests concurrently.

      http://localhost:8080/jdg-test/jdg4
http://localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895
http://localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88

      jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more.

      Siege is available on Fedora (dnf install siege) and can be used by the following command. "urls.txt" has the above contents.

      siege -c 40 -b -f urls.txt
      Show
      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven 2. Deploy it to a standalone EAP 6.4 server. 3. Run the following requests concurrently. http: //localhost:8080/jdg-test/jdg4 http: //localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895 http: //localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88 jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more. Siege is available on Fedora ( dnf install siege ) and can be used by the following command. "urls.txt" has the above contents. siege -c 40 -b -f urls.txt

    Description

      A customer is using a spatial query in a library mode JDG web application. The query sometimes fails with the following exceptions under high load.

      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-3) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index: 181
        at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:158) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]

18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-5) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: java.lang.ArrayIndexOutOfBoundsException: 100
        at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:172) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]

      This is reproducible even on a single server. It seems the error happens if there are concurrent multiple queries with different query parameters. But all objects around LuceneHSQuery.queryEntityInfos() are local to the thread and doesn't look like a thread safety issue at a glance.

      Cache configuration is REPL_ASYNC mode, "ram" directory provider with "near-reql-time" index manager and "async" worker.

      Attachments

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. JDG-1127 "org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. ISPN-7885 "org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is caused by

          HSEARCH-2691 Loading...

          Activity

            People

              gfernand@redhat.com Gustavo Fernandes (Inactive)
              rhn-support-onagano Osamu Nagano
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: