Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-1127

"org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • JDG 7.2 DR1, JDG 7.2 GA
    • JDG 7.0.0 GA, JDG 7.1.0 GA
    • Querying
    • None
    • DR1
    • Hide

      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven
      2. Deploy it to a standalone EAP 6.4 server.
      3. Run the following requests concurrently.

      http://localhost:8080/jdg-test/jdg4
http://localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895
http://localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88

      jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more.

      Siege is available on Fedora (dnf install siege) and can be used by the following command. "urls.txt" has the above contents.

      siege -c 40 -b -f urls.txt
      Show
      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven 2. Deploy it to a standalone EAP 6.4 server. 3. Run the following requests concurrently. http: //localhost:8080/jdg-test/jdg4 http: //localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895 http: //localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88 jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more. Siege is available on Fedora ( dnf install siege ) and can be used by the following command. "urls.txt" has the above contents. siege -c 40 -b -f urls.txt

      A customer is using a spatial query in a library mode JDG web application. The query sometimes fails with the following exceptions under high load.

      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-3) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index: 181
        at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:158) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]

18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-5) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: java.lang.ArrayIndexOutOfBoundsException: 100
        at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:172) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
        at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
        at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]

      This is reproducible even on a single server. It seems the error happens if there are concurrent multiple queries with different query parameters. But all objects around LuceneHSQuery.queryEntityInfos() are local to the thread and doesn't look like a thread safety issue at a glance.

      Cache configuration is REPL_ASYNC mode, "ram" directory provider with "near-reql-time" index manager and "async" worker.

              gfernand@redhat.com Gustavo Fernandes (Inactive)
              rhn-support-onagano Osamu Nagano
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: