Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5026

[Operator] Cluster is unable to start after graceful shutdown on OCP4.9

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • RHDG 8.3 CD
    • None
    • None

    Description

      Node-1 is failing to start up after the upgrade/graceful shutdown with:

      13:20:07,832 FATAL (main) [org.infinispan.SERVER] ISPN080028: Red Hat Data Grid Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
      	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
      	at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:165)
      	at org.infinispan.server.tool.Main.run(Main.java:98)
      	at org.infinispan.server.Bootstrap.main(Bootstrap.java:50)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.infinispan.server.loader.Loader.run(Loader.java:106)
      	at org.infinispan.server.loader.Loader.main(Loader.java:51)
      Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:756)
      	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:720)
      	at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:68)
      	at org.infinispan.security.Security.doPrivileged(Security.java:56)
      	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40)
      	at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:71)
      	at org.infinispan.server.Server.run(Server.java:395)
      	... 9 more
      Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014)
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:700)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:646)
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:535)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:513)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14)
      	at org.infinispan.security.Security.doPrivileged(Security.java:61)
      	at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30)
      	at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39)
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.start(GlobalConfigurationManagerImpl.java:104)
      	at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:60)
      	at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:48)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
      	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:752)
      	... 15 more
      Caused by: java.util.concurrent.CompletionException: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state
      	at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:85)
      	at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:133)
      	at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:104)
      	at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:83)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
      	... 42 more
      Caused by: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state
      	at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:189)
      	at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
      	at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49)
      	at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1451)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1361)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:136)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1502)
      	at org.jgroups.JChannel.up(JChannel.java:784)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
      	at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
      	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048)
      	at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771)
      	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
      	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
      	at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
      	at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:300)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1400)
      	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      
      

      Attachments contain full logs of upgraded servers.

      Attachments

        1. node0
          11 kB
        2. node1
          13 kB
        3. operator.log
          43 kB

        Issue Links

          Activity

            People

              remerson@redhat.com Ryan Emerson
              pdrobek@redhat.com Pavel Drobek
              Pavel Drobek Pavel Drobek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: