Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-7635

Intermittent Indexing incorrectly states Hibernate Search queue is not able to keep up

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • RHDG 8.5.4 GA
    • None
    • Indexing
    • None

      Sometimes indexing will throw an error such as

       at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.lambda$start$0(SearchIndexerImpl.java:68)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl$$Lambda/0x00007deb6f8b4420.accept(Unknown Source:-1)
      	  at io.reactivex.rxjava3.internal.operators.flowable.FlowableOnBackpressureLatest$BackpressureLatestSubscriber.onNext(FlowableOnBackpressureLatest.java:52)
      	  at io.reactivex.rxjava3.processors.UnicastProcessor.drainRegular(UnicastProcessor.java:309)
      	  at io.reactivex.rxjava3.processors.UnicastProcessor.drain(UnicastProcessor.java:384)
      	  at io.reactivex.rxjava3.processors.UnicastProcessor.onNext(UnicastProcessor.java:444)
      	  at io.reactivex.rxjava3.internal.util.NotificationLite.acceptFull(NotificationLite.java:273)
      	  at io.reactivex.rxjava3.internal.util.AppendOnlyLinkedArrayList.accept(AppendOnlyLinkedArrayList.java:119)
      	  at io.reactivex.rxjava3.processors.SerializedProcessor.emitLoop(SerializedProcessor.java:176)
      	  at io.reactivex.rxjava3.processors.SerializedProcessor.onNext(SerializedProcessor.java:104)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.sendOperation(SearchIndexerImpl.java:112)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.lambda$purge$10(SearchIndexerImpl.java:161)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl$$Lambda/0x00007deb6f915408.apply(Unknown Source:-1)
      	  at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:215)
      	  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
      	  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
      	  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:560)
      	  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:636)
      	  at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)
      	  at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:656)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.purge(SearchIndexerImpl.java:165)
      	  at org.infinispan.query.backend.QueryInterceptor.removeFromIndexes(QueryInterceptor.java:384)
      	  at org.infinispan.query.backend.QueryInterceptor.processChange(QueryInterceptor.java:439)
      	  at org.infinispan.query.backend.QueryInterceptor.indexIfNeeded(QueryInterceptor.java:248)
      	  at org.infinispan.query.backend.QueryInterceptor.lambda$handleManyWriteCommand$2(QueryInterceptor.java:215)
      	  at org.infinispan.query.backend.QueryInterceptor$$Lambda/0x00007deb6f7e5ad0.apply(Unknown Source:-1)
      	  at org.infinispan.commons.util.concurrent.CompletionStages.lambda$performConcurrently$3(CompletionStages.java:320)
      	  at org.infinispan.commons.util.concurrent.CompletionStages$$Lambda/0x00007deb6f7e2ef0.apply(Unknown Source:-1)
      	  at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.drain(FlowableConcatMap.java:266)
      	  at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$BaseConcatMapSubscriber.innerComplete(FlowableConcatMap.java:171)
      	  at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapInner.onComplete(FlowableConcatMap.java:587)
      	  at io.reactivex.rxjava3.processors.AsyncProcessor$AsyncSubscription.onComplete(AsyncProcessor.java:359)
      	  at io.reactivex.rxjava3.processors.AsyncProcessor.onComplete(AsyncProcessor.java:197)
      	  at org.infinispan.commons.reactive.RxJavaInterop.lambda$voidCompletionStageToFlowable$0(RxJavaInterop.java:91)
      	  at org.infinispan.commons.reactive.RxJavaInterop$$Lambda/0x00007deb6f919498.accept(Unknown Source:-1)
      	  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
      	  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
      	  at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:554)
      	  at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2223)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl.lambda$sendOperation$2(SearchIndexerImpl.java:95)
      	  at org.infinispan.search.mapper.work.impl.SearchIndexerImpl$$Lambda/0x00007deb6f919260.accept(Unknown Source:-1)
      	  at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:907)
      	  at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:885)
      	  at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:526)
      	  at io.netty.util.concurrent.AbstractEventExecutor.runTask$$$capture(AbstractEventExecutor.java:173)
      	  at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:-1)
      	  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:166)
      	  at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:-1)
      	  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
      	  at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
      	  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
      	  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
       

      However, when reproducing and debugging the queues are not full and each of the processors in each step have requests, showing there is no sign of back pressure issues.

      Further investigating I was able to figure this was related to a specific operator onBackpressureLatest which can present an object as being dropped due to back pressure but is actually due to how the operator was implemented ReactiveX/RxJava#7864. If we use onBackpressureDrop instead it only pays attention to request limits and thus doesn't run into this issue unless there is actually a backpressure issue.

       

              Unassigned Unassigned
              rhn-support-afield Alan Field
              Anna Manukyan Anna Manukyan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: