15:09:08,893 INFO  (jgroups-10,upgrade-restore-2-4-1-8204) [org.infinispan.server.core.backup.BackupManagerImpl] ISPN005045: Starting restore 'upgrade-restore-2-4-1' of '/opt/infinispan/backups/upgrade-backup/upgrade-backup.zip'
15:09:08,910 DEBUG (non-blocking-thread--p2-t5) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Start recover for 'volatileCache' at 1716390548910
15:09:08,911 DEBUG (non-blocking-thread--p2-t6) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Start recover for 'persistentCache' at 1716390548911
15:09:08,933 DEBUG (blocking-thread--p3-t3) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Restoring Cache volatileCache: <?xml version="1.0"?><distributed-cache name="volatileCache" mode="SYNC"/>
15:09:08,933 DEBUG (blocking-thread--p3-t4) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Restoring Cache persistentCache: <?xml version="1.0"?><distributed-cache name="persistentCache" mode="SYNC"><persistence><file-store/></persistence></distributed-cache>
15:09:08,942 WARN  (non-blocking-thread--p2-t1) [org.infinispan.rest.RestRequestHandler] ISPN012026: The REST invocation [NettyRestRequest{HEAD /rest/v2/cache-managers/default/restores/upgrade-restore-2-4-1, remote=/10.244.0.68:59838, subject=null}] has been deprecated. Please consult the upgrade guide
15:09:08,950 WARN  (non-blocking-thread--p2-t1) [org.infinispan.rest.RestRequestHandler] ISPN012026: The REST invocation [NettyRestRequest{HEAD /rest/v2/cache-managers/default/restores/upgrade-restore-2-4-1, remote=/10.244.0.68:59838, subject=null}] has been deprecated. Please consult the upgrade guide
15:09:08,964 DEBUG (non-blocking-thread--p2-t4) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Finished recover for 'volatileCache' at 1716390548963
15:09:08,966 DEBUG (non-blocking-thread--p2-t5) [org.infinispan.server.core.backup.resources.AbstractContainerResource] Finished recover for 'persistentCache' at 1716390548966
15:09:08,969 ERROR (non-blocking-thread--p2-t6) [org.infinispan.server.core.backup.BackupManagerImpl] ISPN005040: Cannot restore cluster backup '/opt/infinispan/backups/upgrade-backup/upgrade-backup.zip': java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'volatileCache' has been requested, but no matching cache configuration exists
	at org.infinispan.commons.util.concurrent.CompletionStages.join(CompletionStages.java:90)
	at org.infinispan.server.core.backup.BackupReader.lambda$restore$0(BackupReader.java:74)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'volatileCache' has been requested, but no matching cache configuration exists
	at org.infinispan.configuration.ConfigurationManager.getConfiguration(ConfigurationManager.java:65)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:682)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:672)
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:565)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:528)
	at org.infinispan.manager.DefaultCacheManagerAdmin.getOrCreateCache(DefaultCacheManagerAdmin.java:50)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:29)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:14)
	at org.infinispan.security.Security.doPrivileged(Security.java:49)
	at org.infinispan.security.actions.SecurityActions.getOrCreateCache(SecurityActions.java:163)
	at org.infinispan.server.core.backup.resources.CacheResource.lambda$createCache$3(CacheResource.java:154)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1495)
	... 1 more

io.reactivex.rxjava3.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at io.reactivex.rxjava3.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:372)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelJoin$JoinSubscription.onError(ParallelJoin.java:192)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelJoin$JoinInnerSubscriber.onError(ParallelJoin.java:508)
	at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:94)
	at io.reactivex.rxjava3.internal.util.HalfSerializer.onError(HalfSerializer.java:68)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.innerError(FlowableConcatMap.java:212)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapInner.onError(FlowableConcatMap.java:575)
	at io.reactivex.rxjava3.processors.AsyncProcessor$AsyncSubscription.onError(AsyncProcessor.java:367)
	at io.reactivex.rxjava3.processors.AsyncProcessor.onError(AsyncProcessor.java:183)
	at org.infinispan.commons.reactive.RxJavaInterop.lambda$voidCompletionStageToFlowable$0(RxJavaInterop.java:89)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1495)
	... 1 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at org.infinispan.configuration.ConfigurationManager.getConfiguration(ConfigurationManager.java:65)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:682)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:672)
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:565)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:528)
	at org.infinispan.manager.DefaultCacheManagerAdmin.getOrCreateCache(DefaultCacheManagerAdmin.java:50)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:29)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:14)
	at org.infinispan.security.Security.doPrivileged(Security.java:49)
	at org.infinispan.security.actions.SecurityActions.getOrCreateCache(SecurityActions.java:163)
	at org.infinispan.server.core.backup.resources.CacheResource.lambda$createCache$3(CacheResource.java:154)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 5 more
Exception in thread "non-blocking-thread--p2-t5" io.reactivex.rxjava3.exceptions.UndeliverableException: The exception could not be delivered to the consumer because it has already canceled/disposed the flow or the exception has nowhere to go to begin with. Further reading: https://github.com/ReactiveX/RxJava/wiki/What's-different-in-2.0#error-handling | java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at io.reactivex.rxjava3.plugins.RxJavaPlugins.onError(RxJavaPlugins.java:372)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelJoin$JoinSubscription.onError(ParallelJoin.java:192)
	at io.reactivex.rxjava3.internal.operators.parallel.ParallelJoin$JoinInnerSubscriber.onError(ParallelJoin.java:508)
	at io.reactivex.rxjava3.internal.util.AtomicThrowable.tryTerminateConsumer(AtomicThrowable.java:94)
	at io.reactivex.rxjava3.internal.util.HalfSerializer.onError(HalfSerializer.java:68)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapImmediate.innerError(FlowableConcatMap.java:212)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatMap$ConcatMapInner.onError(FlowableConcatMap.java:575)
	at io.reactivex.rxjava3.processors.AsyncProcessor$AsyncSubscription.onError(AsyncProcessor.java:367)
	at io.reactivex.rxjava3.processors.AsyncProcessor.onError(AsyncProcessor.java:183)
	at org.infinispan.commons.reactive.RxJavaInterop.lambda$voidCompletionStageToFlowable$0(RxJavaInterop.java:89)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1495)
	... 1 more
Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'persistentCache' has been requested, but no matching cache configuration exists
	at org.infinispan.configuration.ConfigurationManager.getConfiguration(ConfigurationManager.java:65)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:682)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:672)
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:565)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:528)
	at org.infinispan.manager.DefaultCacheManagerAdmin.getOrCreateCache(DefaultCacheManagerAdmin.java:50)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:29)
	at org.infinispan.security.actions.GetOrCreateCacheAction.get(GetOrCreateCacheAction.java:14)
	at org.infinispan.security.Security.doPrivileged(Security.java:49)
	at org.infinispan.security.actions.SecurityActions.getOrCreateCache(SecurityActions.java:163)
	at org.infinispan.server.core.backup.resources.CacheResource.lambda$createCache$3(CacheResource.java:154)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	... 5 more
15:09:08,979 ERROR (jgroups-10,upgrade-restore-2-4-1-8204) [org.infinispan.rest.resources.BackupManagerResource] java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: ISPN005040: Cannot restore cluster backup '/opt/infinispan/backups/upgrade-backup/upgrade-backup.zip'
15:09:09,292 WARN  (non-blocking-thread--p2-t7) [org.infinispan.rest.RestRequestHandler] ISPN012026: The REST invocation [NettyRestRequest{HEAD /rest/v2/cache-managers/default/restores/upgrade-restore-2-4-1, remote=/10.244.0.68:59848, subject=null}] has been deprecated. Please consult the upgrade guide
15:09:09,295 WARN  (non-blocking-thread--p2-t7) [org.infinispan.rest.RestRequestHandler] ISPN012026: The REST invocation [NettyRestRequest{HEAD /rest/v2/cache-managers/default/restores/upgrade-restore-2-4-1, remote=/10.244.0.68:59848, subject=null}] has been deprecated. Please consult the upgrade guide
15:09:09,296 ERROR (non-blocking-thread--p2-t1) [org.infinispan.rest.RestRequestHandler] ISPN012005: An error occurred while responding to the client java.lang.IllegalStateException: ISPN012039: Restore failed
	at org.infinispan.rest.resources.BackupManagerResource.handleRestoreStatus(BackupManagerResource.java:151)
	at org.infinispan.rest.resources.BackupManagerResource.handleRestoreRequest(BackupManagerResource.java:129)
	at org.infinispan.rest.resources.ContainerResource.restore(ContainerResource.java:375)
	at org.infinispan.security.Security.doAs(Security.java:114)
	at org.infinispan.rest.framework.impl.RestDispatcherImpl.dispatch(RestDispatcherImpl.java:83)
	at org.infinispan.rest.RestRequestHandler.handleRestRequest(RestRequestHandler.java:208)
	at org.infinispan.rest.RestRequestHandler.lambda$channelRead0$1(RestRequestHandler.java:187)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1583)