Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-13131

REST keys operation blocks in a non-blocking thread

    XMLWordPrintable

    Details

      Description

      CaceResourceV2.streamKeys returns a CacheKeyInputStream, which is converted to a chunked HTTP response in
      BaseHttpRequestHandler.sendResponse().

      CacheKeyInputStream uses cache streams internally, which block when some of the keys are not local:

      Test failed: org.infinispan.rest.resources.Test[security=true, protocol=HTTP_20, ssl=true].BlockingChecker
      java.lang.AssertionError: Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[non-blocking-thread-Test-NodeA-p1147-t1,5,ISPN-non-blocking-thread-group]
      	at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:52) ~[infinispan-core-13.0.0-SNAPSHOT-tests.jar:13.0.0-SNAPSHOT]
      	at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:383) ~[blockhound-1.0.3.RELEASE.jar:?]
      	at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89) ~[?:?]
      	at jdk.internal.misc.Unsafe.park(Unsafe.java) ~[?:?]
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) ~[?:?]
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) ~[?:?]
      	at io.reactivex.rxjava3.internal.operators.flowable.BlockingFlowableIterable$BlockingFlowableIterator.hasNext(BlockingFlowableIterable.java:101) ~[rxjava-3.0.4.jar:?]
      	at org.infinispan.commons.util.Closeables$1.hasNext(Closeables.java:247) ~[infinispan-commons-13.0.0-SNAPSHOT.jar:13.0.0-SNAPSHOT]
      	at org.infinispan.rest.CacheKeyInputStream.read(CacheKeyInputStream.java:69) ~[classes/:?]
      	at java.io.InputStream.read(InputStream.java:280) ~[?:?]
      	at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[?:?]
      	at java.io.PushbackInputStream.read(PushbackInputStream.java:183) ~[?:?]
      	at io.netty.buffer.UnsafeByteBufUtil.setBytes(UnsafeByteBufUtil.java:454) ~[netty-buffer-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:211) ~[netty-buffer-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1122) ~[netty-buffer-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.stream.ChunkedStream.readChunk(ChunkedStream.java:122) ~[netty-handler-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.stream.ChunkedStream.readChunk(ChunkedStream.java:36) ~[netty-handler-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:102) ~[netty-codec-http-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:43) ~[netty-codec-http-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:230) ~[netty-handler-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132) ~[netty-handler-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750) ~[netty-transport-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765) ~[netty-transport-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[netty-transport-4.1.63.Final.jar:4.1.63.Final]
      	at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[netty-transport-4.1.63.Final.jar:4.1.63.Final]
      	at org.infinispan.rest.ResponseWriter$4.writeResponse(ResponseWriter.java:85) ~[classes/:?]
      	at org.infinispan.rest.BaseHttpRequestHandler.lambda$sendResponse$0(BaseHttpRequestHandler.java:47) ~[classes/:?]
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.63.Final.jar:4.1.63.Final]
      

      The blocking operation causes random failures in CacheResourceTest and CacheV2ResourceTest.
      Increasing the cluster size in AbstractRestResourceTest (e.g. to 12) makes the failure much easier to reproduce.

      Test failed: org.infinispan.rest.resources.CacheResourceTest.shouldGetAllEntriesConvertedToJson[security=true, protocol=HTTP_20, ssl=true]
      java.lang.NullPointerException: null
      	at org.infinispan.rest.assertion.ResponseAssertion.lambda$hasReturnedText$0(ResponseAssertion.java:98) ~[test-classes/:?]
      	at org.assertj.core.api.AbstractAssert.matches(AbstractAssert.java:528) ~[assertj-core-3.4.1.jar:?]
      	at org.assertj.core.api.AbstractAssert.matches(AbstractAssert.java:523) ~[assertj-core-3.4.1.jar:?]
      	at org.infinispan.rest.assertion.ResponseAssertion.hasReturnedText(ResponseAssertion.java:96) ~[test-classes/:?]
      	at org.infinispan.rest.resources.BaseCacheResourceTest.shouldGetAllEntriesConvertedToJson(BaseCacheResourceTest.java:392) ~[test-classes/:?]
      
      Test failed: org.infinispan.rest.resources.CacheV2ResourceTest.testGetAllKeys[security=true, protocol=HTTP_20, ssl=true]
      java.lang.NullPointerException
      	at java.base/java.text.StringCharacterIterator.<init>(StringCharacterIterator.java:79)
      	at java.base/java.text.StringCharacterIterator.<init>(StringCharacterIterator.java:68)
      	at org.infinispan.commons.dataconversion.internal.Json$Reader.read(Json.java:3045)
      	at org.infinispan.commons.dataconversion.internal.Json.read(Json.java:1312)
      	at org.infinispan.rest.resources.CacheV2ResourceTest.testGetAllKeys(CacheV2ResourceTest.java:494)
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dan.berindei Dan Berindei
              Reporter:
              dan.berindei Dan Berindei
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: