Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-13116

SFS failures on upgrade from 11.0.9.Final -> 12.1.3.Final

XMLWordPrintable

    • Hide
      # Init 11.x server
      cd /tmp
      unzip infinispan-server-11.0.9.Final.zip
      ./infinispan-server-11.0.9.Final/bin/cli.sh user create admin -p pass
      cp -r infinispan-server-11.0.9.Final/server /tmp/server
      
      # Start 11.x server
      ./infinispan-server-11.0.9.Final/bin/server.sh -s /tmp/server
      
      # New terminal connect CLI and Add cache configuriation with SFS and add some values
      ./infinispan-server-11.0.9.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
      create cache --file=/tmp/cache.xml update-cache
      cd caches/update-cache
      put update-key update-value
      ls
      
      # Stop 11.x server
      ctrl+c
      
      # Init 12.x server
      unzip infinispan-server-12.1.4.Final.zip
      ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server
      
      # Can read entries fine
      ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
      ls caches/update-cache
      
      # Restart 12.1.x server after successful startup
      ctrl+c
      ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server
      
      # Unable to read entries
      ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
      ls caches/update-cache
      "Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114"
      
      Show
      # Init 11.x server cd /tmp unzip infinispan-server-11.0.9.Final.zip ./infinispan-server-11.0.9.Final/bin/cli.sh user create admin -p pass cp -r infinispan-server-11.0.9.Final/server /tmp/server # Start 11.x server ./infinispan-server-11.0.9.Final/bin/server.sh -s /tmp/server # New terminal connect CLI and Add cache configuriation with SFS and add some values ./infinispan-server-11.0.9.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 create cache --file=/tmp/cache.xml update-cache cd caches/update-cache put update-key update-value ls # Stop 11.x server ctrl+c # Init 12.x server unzip infinispan-server-12.1.4.Final.zip ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server # Can read entries fine ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 ls caches/update-cache # Restart 12.1.x server after successful startup ctrl+c ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server # Unable to read entries ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 ls caches/update-cache "Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114"
    • Undefined

      Upgrading caches containing a SFS from 11.0.9.Final -> 12.1.4.Final works as expected (11.0.9.Final.logs) during the first startup of the 12.1.4.Final server (12.1.4.Final-upgrade.logs). Persisted entries are readable at runtime without issue. However, if the 12.1.4.Final server is subsequentally restarted (12.1.4.Final-restart.logs), it's no longer possible to read the cache entries in the SFS. When trying to retrieve a cache entry the following error is thrown:

      Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114
      

      The following error is also thrown on Server shutdown.

      12:29:31,490 WARN  (Thread-0) [org.infinispan.CONTAINER] ISPN000189: While stopping a cache or cache manager, one of its components failed to stop java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
      	at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412)
      	at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044)
      	at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:781)
      	at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:842)
      	at org.infinispan.manager.DefaultCacheManager.internalStop(DefaultCacheManager.java:818)
      	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:794)
      	at org.infinispan.server.SecurityActions.lambda$stopCacheManager$2(SecurityActions.java:76)
      	at org.infinispan.security.Security.doPrivileged(Security.java:56)
      	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
      	at org.infinispan.server.SecurityActions.stopCacheManager(SecurityActions.java:82)
      	at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
      	at org.infinispan.server.Server.localShutdown(Server.java:505)
      	at org.infinispan.server.Server.lambda$run$1(Server.java:424)
      	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
      	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
      	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
      	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
      	at org.infinispan.server.DefaultExitHandler.exit(DefaultExitHandler.java:13)
      	at org.infinispan.server.ShutdownHook.run(ShutdownHook.java:16)
      Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:572)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:213)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1015)
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:698)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:644)
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:533)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:511)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14)
      	at org.infinispan.security.Security.doPrivileged(Security.java:56)
      	at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30)
      	at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39)
      	at org.infinispan.globalstate.impl.VolatileLocalConfigurationStorage.lambda$createCache$0(VolatileLocalConfigurationStorage.java:86)
      	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: java.util.concurrent.CompletionException: org.infinispan.persistence.spi.PersistenceException: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114
      	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
      	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
      	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
      	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
      	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
      	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:143)
      	at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$start$2(PersistenceManagerImpl.java:203)
      	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.drain(FlowableConcatMapSingle.java:249)
      	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.onNext(FlowableConcatMapSingle.java:136)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:243)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:131)
      	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.onSubscribe(FlowableConcatMapSingle.java:125)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
      	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
      	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle.subscribeActual(FlowableConcatMapSingle.java:61)
      	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
      	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:52)
      	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
      	at io.reactivex.rxjava3.internal.operators.flowable.FlowableIgnoreElementsCompletable.subscribeActual(FlowableIgnoreElementsCompletable.java:34)
      	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
      	at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
      	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
      	at io.reactivex.rxjava3.core.Completable.blockingAwait(Completable.java:1460)
      	at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:230)
      	at org.infinispan.persistence.manager.CorePackageImpl$3.start(CorePackageImpl.java:87)
      	at org.infinispan.persistence.manager.CorePackageImpl$3.start(CorePackageImpl.java:64)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
      	... 38 more
      Caused by: org.infinispan.persistence.spi.PersistenceException: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114
      	at org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:236)
      	at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:162)
      	at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.startNewStoreForSegment(ComposedSegmentedLoadWriteStore.java:254)
      	at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.start(ComposedSegmentedLoadWriteStore.java:239)
      	at org.infinispan.persistence.support.NonBlockingStoreAdapter.lambda$start$0(NonBlockingStoreAdapter.java:107)
      	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlockingOperation(BlockingManagerImpl.java:109)
      	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlockingOperation(BlockingManagerImpl.java:99)
      	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlocking(BlockingManagerImpl.java:65)
      	at org.infinispan.persistence.support.NonBlockingStoreAdapter.start(NonBlockingStoreAdapter.java:101)
      	... 64 more
      
      

              remerson@redhat.com Ryan Emerson
              remerson@redhat.com Ryan Emerson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: