Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-13116

SFS failures on upgrade from 11.0.9.Final -> 12.1.3.Final

    XMLWordPrintable

Details

    • Hide 
      # Init 11.x server
cd /tmp
unzip infinispan-server-11.0.9.Final.zip
./infinispan-server-11.0.9.Final/bin/cli.sh user create admin -p pass
cp -r infinispan-server-11.0.9.Final/server /tmp/server

# Start 11.x server
./infinispan-server-11.0.9.Final/bin/server.sh -s /tmp/server

# New terminal connect CLI and Add cache configuriation with SFS and add some values
./infinispan-server-11.0.9.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
create cache --file=/tmp/cache.xml update-cache
cd caches/update-cache
put update-key update-value
ls

# Stop 11.x server
ctrl+c

# Init 12.x server
unzip infinispan-server-12.1.4.Final.zip
./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server

# Can read entries fine
./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
ls caches/update-cache

# Restart 12.1.x server after successful startup
ctrl+c
./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server

# Unable to read entries
./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222
ls caches/update-cache
"Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114"
      Show
      # Init 11.x server cd /tmp unzip infinispan-server-11.0.9.Final.zip ./infinispan-server-11.0.9.Final/bin/cli.sh user create admin -p pass cp -r infinispan-server-11.0.9.Final/server /tmp/server # Start 11.x server ./infinispan-server-11.0.9.Final/bin/server.sh -s /tmp/server # New terminal connect CLI and Add cache configuriation with SFS and add some values ./infinispan-server-11.0.9.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 create cache --file=/tmp/cache.xml update-cache cd caches/update-cache put update-key update-value ls # Stop 11.x server ctrl+c # Init 12.x server unzip infinispan-server-12.1.4.Final.zip ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server # Can read entries fine ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 ls caches/update-cache # Restart 12.1.x server after successful startup ctrl+c ./infinispan-server-12.1.4.Final/bin/server.sh -s /tmp/server # Unable to read entries ./infinispan-server-12.1.4.Final/bin/cli.sh -c http://admin:pass@127.0.0.1:11222 ls caches/update-cache "Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114"
    • Undefined

    Description

      Upgrading caches containing a SFS from 11.0.9.Final -> 12.1.4.Final works as expected (11.0.9.Final.logs) during the first startup of the 12.1.4.Final server (12.1.4.Final-upgrade.logs). Persisted entries are readable at runtime without issue. However, if the 12.1.4.Final server is subsequentally restarted (12.1.4.Final-restart.logs), it's no longer possible to read the cache entries in the SFS. When trying to retrieve a cache entry the following error is thrown:

      Error: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114

      The following error is also thrown on Server shutdown.

      12:29:31,490 WARN  (Thread-0) [org.infinispan.CONTAINER] ISPN000189: While stopping a cache or cache manager, one of its components failed to stop java.util.concurrent.CompletionException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
	at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412)
	at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044)
	at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:781)
	at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:842)
	at org.infinispan.manager.DefaultCacheManager.internalStop(DefaultCacheManager.java:818)
	at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:794)
	at org.infinispan.server.SecurityActions.lambda$stopCacheManager$2(SecurityActions.java:76)
	at org.infinispan.security.Security.doPrivileged(Security.java:56)
	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:39)
	at org.infinispan.server.SecurityActions.stopCacheManager(SecurityActions.java:82)
	at java.base/java.util.LinkedHashMap$LinkedValues.forEach(LinkedHashMap.java:608)
	at org.infinispan.server.Server.localShutdown(Server.java:505)
	at org.infinispan.server.Server.lambda$run$1(Server.java:424)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at org.infinispan.server.DefaultExitHandler.exit(DefaultExitHandler.java:13)
	at org.infinispan.server.ShutdownHook.run(ShutdownHook.java:16)
Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.persistence.manager.PersistenceManager
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:572)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:622)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:586)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:787)
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:213)
	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1015)
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:698)
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:644)
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:533)
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:511)
	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26)
	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14)
	at org.infinispan.security.Security.doPrivileged(Security.java:56)
	at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30)
	at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39)
	at org.infinispan.globalstate.impl.VolatileLocalConfigurationStorage.lambda$createCache$0(VolatileLocalConfigurationStorage.java:86)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.CompletionException: org.infinispan.persistence.spi.PersistenceException: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
	at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:143)
	at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$start$2(PersistenceManagerImpl.java:203)
	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.drain(FlowableConcatMapSingle.java:249)
	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.onNext(FlowableConcatMapSingle.java:136)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:243)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:131)
	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle$ConcatMapSingleSubscriber.onSubscribe(FlowableConcatMapSingle.java:125)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.mixed.FlowableConcatMapSingle.subscribeActual(FlowableConcatMapSingle.java:61)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableDoOnEach.subscribeActual(FlowableDoOnEach.java:50)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap.subscribeActual(FlowableFlatMap.java:52)
	at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750)
	at io.reactivex.rxjava3.internal.operators.flowable.FlowableIgnoreElementsCompletable.subscribeActual(FlowableIgnoreElementsCompletable.java:34)
	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
	at io.reactivex.rxjava3.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
	at io.reactivex.rxjava3.core.Completable.subscribe(Completable.java:2850)
	at io.reactivex.rxjava3.core.Completable.blockingAwait(Completable.java:1460)
	at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:230)
	at org.infinispan.persistence.manager.CorePackageImpl$3.start(CorePackageImpl.java:87)
	at org.infinispan.persistence.manager.CorePackageImpl$3.start(CorePackageImpl.java:64)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:604)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:595)
	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:564)
	... 38 more
Caused by: org.infinispan.persistence.spi.PersistenceException: ISPN000279: Failed to read stored entries from file. Error in file /tmp/server/data/184/update-cache.dat at offset 114
	at org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:236)
	at org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:162)
	at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.startNewStoreForSegment(ComposedSegmentedLoadWriteStore.java:254)
	at org.infinispan.persistence.support.ComposedSegmentedLoadWriteStore.start(ComposedSegmentedLoadWriteStore.java:239)
	at org.infinispan.persistence.support.NonBlockingStoreAdapter.lambda$start$0(NonBlockingStoreAdapter.java:107)
	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlockingOperation(BlockingManagerImpl.java:109)
	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlockingOperation(BlockingManagerImpl.java:99)
	at org.infinispan.util.concurrent.BlockingManagerImpl.runBlocking(BlockingManagerImpl.java:65)
	at org.infinispan.persistence.support.NonBlockingStoreAdapter.start(NonBlockingStoreAdapter.java:101)
	... 64 more

      Attachments

        Activity

          People

            remerson@redhat.com Ryan Emerson
            remerson@redhat.com Ryan Emerson
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: