-
Bug
-
Resolution: Done
-
Blocker
-
RHDG 8.3 CD
-
None
-
None
Node-1 is failing to start up after the upgrade/graceful shutdown with:
13:20:07,832 FATAL (main) [org.infinispan.SERVER] ISPN080028: Red Hat Data Grid Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:165) at org.infinispan.server.tool.Main.run(Main.java:98) at org.infinispan.server.Bootstrap.main(Bootstrap.java:50) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.server.loader.Loader.run(Loader.java:106) at org.infinispan.server.loader.Loader.main(Loader.java:51) Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:756) at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:720) at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:68) at org.infinispan.security.Security.doPrivileged(Security.java:56) at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40) at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:71) at org.infinispan.server.Server.run(Server.java:395) ... 9 more Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:700) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:646) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:535) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:513) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14) at org.infinispan.security.Security.doPrivileged(Security.java:61) at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30) at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39) at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.start(GlobalConfigurationManagerImpl.java:104) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:60) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:48) at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:752) ... 15 more Caused by: java.util.concurrent.CompletionException: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:85) at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:133) at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:104) at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:83) at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) ... 42 more Caused by: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:189) at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1451) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1361) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:136) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1502) at org.jgroups.JChannel.up(JChannel.java:784) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913) at org.jgroups.protocols.FRAG3.up(FRAG3.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Attachments contain full logs of upgraded servers.