-
Bug
-
Resolution: Done
-
Blocker
-
RHDG 8.3 CD
-
None
Second node fails to join the cluster after upgrade from 8.3.3.CSV to 8.3.4.CSV
$ oc logs test-upgrade-1 13:26:17,419 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.14.1+1-LTS 13:26:17,423 INFO (main) [BOOT] JVM arguments = [-server, -Dinfinispan.zero-capacity-node=false, -Xmx512m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=redhat-datagrid-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-13.0.9.CD20220407-temporary-redhat-00001-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap, --bind-address=0.0.0.0, -l, /opt/infinispan/server/conf/operator/log4j.xml, -c, operator/infinispan.xml] 13:26:17,423 INFO (main) [BOOT] PID = 137 13:26:17,452 INFO (main) [org.infinispan.SERVER] ISPN080000: Red Hat Data Grid Server starting 13:26:17,453 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: operator/infinispan.xml 13:26:17,453 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/operator/log4j.xml 13:26:17,756 WARN (main) [org.infinispan.CONFIG] ISPN000653: The 'keystore-password' attribute on the 'keystore' element has been deprecated. Please use the 'password' attribute instead 13:26:17,822 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 13:26:17,822 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 13:26:17,824 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory' 13:26:17,825 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory' 13:26:17,887 INFO (main) [org.wildfly.openssl.SSL] WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1k FIPS 25 Mar 2021 13:26:17,888 INFO (main) [org.infinispan.SECURITY] ISPN000946: Using OpenSSL Provider 13:26:18,440 WARN (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal 13:26:18,459 INFO (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller' 13:26:18,833 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Red Hat Data Grid 'Infinispan 13.0.9.CD20220407-temporary-redhat-00001' 8.3.1.GA 13:26:18,834 INFO (main) [org.infinispan.CONTAINER] ISPN000389: Loaded global state, version=13.0.9.CD20220407-temporary-redhat-00001 timestamp=2022-04-11T13:26:13.987240Z 13:26:18,936 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel `` with stack `image-tcp` 13:26:19,070 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel : [test-upgrade-0-7887|3] (2) [test-upgrade-0-7887, test-upgrade-1-56943] 13:26:19,113 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel `` local address is `test-upgrade-1-56943`, physical addresses are `[10.129.3.238:7800]` 13:26:19,317 INFO (main) [org.jboss.threads] JBoss Threads version 2.3.3.Final-redhat-00001 13:26:19,364 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 13:26:19,429 INFO (main) [org.infinispan.CLUSTER] ISPN000080: Disconnecting JGroups channel `` 13:26:19,439 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=13.0.9.CD20220407-temporary-redhat-00001 timestamp=2022-04-11T13:26:19.436586Z 13:26:19,546 FATAL (main) [org.infinispan.SERVER] ISPN080028: Red Hat Data Grid Server failed to start org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:770) at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:734) at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:69) at org.infinispan.security.Security.doPrivileged(Security.java:56) at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40) at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:72) at org.infinispan.server.Server.run(Server.java:398) at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:165) at org.infinispan.server.tool.Main.run(Main.java:98) at org.infinispan.server.Bootstrap.main(Bootstrap.java:50) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.server.loader.Loader.run(Loader.java:106) at org.infinispan.server.loader.Loader.main(Loader.java:51) Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:714) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:660) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:549) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:517) at org.infinispan.server.hotrod.LifecycleCallbacks.createGlobalTxTable(LifecycleCallbacks.java:147) at org.infinispan.server.hotrod.LifecycleCallbacks.registerServerTransactionTable(LifecycleCallbacks.java:119) at org.infinispan.server.hotrod.LifecycleCallbacks.cacheStarting(LifecycleCallbacks.java:106) at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:239) at org.infinispan.factories.ComponentRegistry.preStart(ComponentRegistry.java:226) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:248) at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216) at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014) at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:714) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:660) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:549) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:517) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26) at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14) at org.infinispan.security.Security.doPrivileged(Security.java:61) at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30) at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39) at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.start(GlobalConfigurationManagerImpl.java:104) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:60) at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:48) at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30) at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808) at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354) at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250) at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:766) ... 15 more Caused by: java.util.concurrent.CompletionException: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-56943 with persistent state attempting to join cache org.infinispan.CLIENT_SERVER_TX_TABLE on cluster without state at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:85) at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:133) at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:104) at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:83) at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617) at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608) at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577) ... 57 more Caused by: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-56943 with persistent state attempting to join cache org.infinispan.CLIENT_SERVER_TX_TABLE on cluster without state at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:189) at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1487) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1397) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:145) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1582) at org.jgroups.JChannel.up(JChannel.java:780) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913) at org.jgroups.protocols.FRAG3.up(FRAG3.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
- links to