Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5544

Potential deadlock in index writer threads under heavy load

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • RHDG 8.4.1 GA
    • RHDG 8.3.1 GA
    • Querying
    • False
    • None
    • False
    • Hide

      Raising the precessing queue makes the deadlock less likely

         <index-writer queue-count="10" queue-size="10000"/>{}

       

      Show
      Raising the precessing queue makes the deadlock less likely    <index-writer queue-count="10" queue-size="10000"/> { }  

      Originally reported on EAP: https://issues.redhat.com/browse/EAPSUP-961

      In some cases, it seems Infinispan will try to follow-up an indexing operations by another.

      See https://github.com/infinispan/infinispan/blob/7bd0ed0b68eaa48b8b6e15496316da54914cc895/query/src/main/java/org/infinispan/query/backend/QueryInterceptor.java#L433

      The request to update indexes is added in thenCompose(), which means it's added as soon as the first operation is marked as complete, which actually happens in the index writer thread... which is precisely the thread responsible for consuming that queue.

      That can lead to a deadlock: should the queue be full, the act of adding an element to that queue will block, which means the index writer thread, responsible for consuming that queue, will block, which means the queue will never get processed ever again.

      The bug seems to have been introduced in ISPN-11731: https://github.com/infinispan/infinispan/commit/c68c255f3b81dfcb71a25051b38be94700ad5ef5

              fercoli@redhat.com Fabio Massimo Ercoli
              rh-ee-yrodiere Yoann Rodière
              Diego Lovison Diego Lovison
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: