Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-1002

"org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index" under high load

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: JDG 7.0.0 GA, JDG 7.1.0 GA
    • Fix Version/s: JDG 7.0.1
    • Component/s: Querying
    • Labels:
      None
    • Target Release:
    • Fix Build:
      DR2
    • Steps to Reproduce:
      Hide

      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven
      2. Deploy it to a standalone EAP 6.4 server.
      3. Run the following requests concurrently.

      http://localhost:8080/jdg-test/jdg4
      http://localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895
      http://localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88
      

      jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more.

      Siege is available on Fedora (dnf install siege) and can be used by the following command. "urls.txt" has the above contents.

      siege -c 40 -b -f urls.txt
      
      Show
      1. Unzip and build the attached project, jdg-test.zip. Use this settings for maven 2. Deploy it to a standalone EAP 6.4 server. 3. Run the following requests concurrently. http: //localhost:8080/jdg-test/jdg4 http: //localhost:8080/jdg-test/spatialQuery?radius=10000&x=103.285&y=10.7895 http: //localhost:8080/jdg-test/spatialQuery?radius=3000&x=103.285&y=10.88 jdg4 puts a new entry and spatialQuery executes a spatial query. The error will happen in several minutes using JMeter or Siege with 40 threads or more. Siege is available on Fedora ( dnf install siege ) and can be used by the following command. "urls.txt" has the above contents. siege -c 40 -b -f urls.txt

      Description

      A customer is using a spatial query in a library mode JDG web application. The query sometimes fails with the following exceptions under high load.

      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-3) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: org.hibernate.search.exception.SearchException: Not a valid ScoreDoc index: 181
              at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:158) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
      
      18:02:00,415 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/jdg-test].[com.myapp.JDGTest5]] (http-/127.0.0.1:8080-5) JBWEB000236: Servlet.service() for servlet com.myapp.JDGTest5 threw exception: java.lang.ArrayIndexOutOfBoundsException: 100
              at org.hibernate.search.query.engine.impl.QueryHits.scoreDoc(QueryHits.java:172) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.QueryHits.docId(QueryHits.java:176) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.DocumentExtractorImpl.extract(DocumentExtractorImpl.java:185) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.hibernate.search.query.engine.impl.LuceneHSQuery.queryEntityInfos(LuceneHSQuery.java:159) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at org.infinispan.query.impl.CacheQueryImpl.list(CacheQueryImpl.java:160) [infinispan-embedded-query-8.4.0.Final-redhat-2.jar:8.4.0.Final-redhat-2]
              at com.myapp.JDGTest5.doGet(JDGTest5.java:72) [classes:]
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
              at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.2.Final-redhat-2.jar:1.0.2.Final-redhat-2]
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
              at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb-7.5.10.Final-redhat-1.jar:7.5.10.Final-redhat-1]
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_131]
      

      This is reproducible even on a single server. It seems the error happens if there are concurrent multiple queries with different query parameters. But all objects around LuceneHSQuery.queryEntityInfos() are local to the thread and doesn't look like a thread safety issue at a glance.

      Cache configuration is REPL_ASYNC mode, "ram" directory provider with "near-reql-time" index manager and "async" worker.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  gustavonalle Gustavo Fernandes
                  Reporter:
                  osamu.nagano Osamu Nagano
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: