-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
RHDG 8.6.x CD
-
False
-
-
False
-
-
Please note, the issue is for Rolling Upgrades using Remote Store method available during DG lifetime.
For Rolling upgrades from DG 8.5.5 and DG 8.4.8:
For these servers, the RU works properly when the cache store/cache config has specified the encoding. In case the encoding is not specified (and I am not sure in that case which is the default), the migration process throws an exception on 8.6.0 server:
io.reactivex.rxjava3.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | org.infinispan.client.hotrod.exceptions.HotRodClientException:: ISPN004034: Unable to unmarshall bytes stream at io.reactivex.rxjava3.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:372) at io.reactivex.rxjava3.processors.UnicastProcessor.onError(UnicastProcessor.java:452) at org.infinispan.commons.reactive.AbstractAsyncPublisherHandler.handleThrowableInResponse(AbstractAsyncPublisherHandler.java:133) at org.infinispan.client.hotrod.impl.iteration.RemoteInnerPublisherHandler.handleThrowableInResponse(RemoteInnerPublisherHandler.java:139) at org.infinispan.client.hotrod.impl.iteration.RemoteInnerPublisherHandler.handleThrowableInResponse(RemoteInnerPublisherHandler.java:23) at org.infinispan.commons.reactive.AbstractAsyncPublisherHandler$ResponseConsumer.accept(AbstractAsyncPublisherHandler.java:200) at org.infinispan.commons.reactive.AbstractAsyncPublisherHandler$ResponseConsumer.accept(AbstractAsyncPublisherHandler.java:195) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.checkException(OperationDispatcher.java:760) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.handleResponse(OperationDispatcher.java:688) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.handleResponse(OperationDispatcher.java:679) at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:296) at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:94) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.infinispan.client.hotrod.impl.transport.netty.IdleStateHandlerNoUnvoid.channelRead(IdleStateHandlerNoUnvoid.java:285) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799) at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: org.infinispan.client.hotrod.exceptions.HotRodClientException:: ISPN004034: Unable to unmarshall bytes stream at org.infinispan.client.hotrod.marshall.MarshallerUtil.bytes2obj(MarshallerUtil.java:102) at org.infinispan.client.hotrod.DataFormat$DataFormatImpl.bytesToValue(DataFormat.java:134) at org.infinispan.client.hotrod.DataFormat.valueToObj(DataFormat.java:244) at org.infinispan.client.hotrod.impl.operations.ByteBufCacheUnmarshaller.readValue(ByteBufCacheUnmarshaller.java:64) at org.infinispan.client.hotrod.impl.operations.ByteBufCacheUnmarshaller.readValue(ByteBufCacheUnmarshaller.java:50) at org.infinispan.client.hotrod.impl.operations.IterationNextOperation.createResponse(IterationNextOperation.java:102) at org.infinispan.client.hotrod.impl.operations.IterationNextOperation.createResponse(IterationNextOperation.java:29) at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:291) ... 22 more Suppressed: org.infinispan.commons.util.logging.TraceException at org.infinispan.client.hotrod.impl.iteration.RemoteInnerPublisherHandler.handleThrowableInResponse(RemoteInnerPublisherHandler.java:138) at org.infinispan.client.hotrod.impl.iteration.RemoteInnerPublisherHandler.handleThrowableInResponse(RemoteInnerPublisherHandler.java:23) at org.infinispan.commons.reactive.AbstractAsyncPublisherHandler$ResponseConsumer.accept(AbstractAsyncPublisherHandler.java:200) at org.infinispan.commons.reactive.AbstractAsyncPublisherHandler$ResponseConsumer.accept(AbstractAsyncPublisherHandler.java:195) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.checkException(OperationDispatcher.java:760) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.handleResponse(OperationDispatcher.java:688) at org.infinispan.client.hotrod.impl.transport.netty.OperationDispatcher.handleResponse(OperationDispatcher.java:679) at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:296) ... 22 more
You can find the full log in attachment 8.5.5-8.6.server.log .
Other caches migration like eviction (encoding is text/plain) or protubuff (encoding is application/x-protostream) works properly.
*For Rolling upgrades from DG 8.1.x - DG 8.3.1 : *
In this case the migration fails in all cases. Please find attached the server log : 8.3.1-8.6.server.log
*For Rolling upgrades from DG 8.0.1: *
Here again all caches migration fails but with some kind of other exceptions. Please find attached: 8.0.1-8.6.server.log
You can find the reproducer here: https://gitlab.cee.redhat.com/dg-qe/dg-qe/-/tree/master/scripts/rolling-upgrades?ref_type=heads
For running it, download DG 8.6.0.nightly build and the version of the server which you need for testing, and the run the following command:
bash -x ./do-rolling.sh -s /path/to/source/server/ -t /path/to/8.6.0/server/
In case of successfull run you will find in logs the following lines (mixed with other logs):
default cache - TEST PASSED
eviction cache - TEST PASSED
protobuff cache - TEST PASSED
