Uploaded image for project: 'Red Hat Data Grid'
  1. Red Hat Data Grid
  2. JDG-5026

[Operator] Cluster is unable to start after graceful shutdown on OCP4.9

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • RHDG 8.3 CD
    • None
    • None

    Description

      Node-1 is failing to start up after the upgrade/graceful shutdown with:

      13:20:07,832 FATAL (main) [org.infinispan.SERVER] ISPN080028: Red Hat Data Grid Server failed to start java.util.concurrent.ExecutionException: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
      	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
      	at org.infinispan.server.Bootstrap.runInternal(Bootstrap.java:165)
      	at org.infinispan.server.tool.Main.run(Main.java:98)
      	at org.infinispan.server.Bootstrap.main(Bootstrap.java:50)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.infinispan.server.loader.Loader.run(Loader.java:106)
      	at org.infinispan.server.loader.Loader.main(Loader.java:51)
      Caused by: org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:756)
      	at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:720)
      	at org.infinispan.server.SecurityActions.lambda$startCacheManager$1(SecurityActions.java:68)
      	at org.infinispan.security.Security.doPrivileged(Security.java:56)
      	at org.infinispan.server.SecurityActions.doPrivileged(SecurityActions.java:40)
      	at org.infinispan.server.SecurityActions.startCacheManager(SecurityActions.java:71)
      	at org.infinispan.server.Server.run(Server.java:395)
      	... 9 more
      Caused by: org.infinispan.commons.CacheConfigurationException: Error starting component org.infinispan.statetransfer.StateTransferManager
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1014)
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:700)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:646)
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:535)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:513)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26)
      	at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14)
      	at org.infinispan.security.Security.doPrivileged(Security.java:61)
      	at org.infinispan.globalstate.impl.SecurityActions.doPrivileged(SecurityActions.java:30)
      	at org.infinispan.globalstate.impl.SecurityActions.getCache(SecurityActions.java:39)
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.start(GlobalConfigurationManagerImpl.java:104)
      	at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:60)
      	at org.infinispan.globalstate.impl.CorePackageImpl$2.start(CorePackageImpl.java:48)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.access$700(BasicComponentRegistryImpl.java:30)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:354)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
      	at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:752)
      	... 15 more
      Caused by: java.util.concurrent.CompletionException: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state
      	at org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:85)
      	at org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:133)
      	at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:104)
      	at org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:83)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
      	at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
      	... 42 more
      Caused by: org.infinispan.topology.CacheJoinException: ISPN000408: Node test-upgrade-1-41920 with persistent state attempting to join cache ___protobuf_metadata on cluster without state
      	at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:189)
      	at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32)
      	at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49)
      	at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41)
      	at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708)
      	at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191)
      	at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1451)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1361)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:136)
      	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1502)
      	at org.jgroups.JChannel.up(JChannel.java:784)
      	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913)
      	at org.jgroups.protocols.FRAG3.up(FRAG3.java:165)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
      	at org.jgroups.protocols.FlowControl.up(FlowControl.java:347)
      	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876)
      	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254)
      	at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048)
      	at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771)
      	at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752)
      	at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405)
      	at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592)
      	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132)
      	at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186)
      	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254)
      	at org.jgroups.protocols.MERGE3.up(MERGE3.java:281)
      	at org.jgroups.protocols.Discovery.up(Discovery.java:300)
      	at org.jgroups.protocols.TP.passMessageUp(TP.java:1400)
      	at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      
      

      Attachments contain full logs of upgraded servers.

      Attachments

        1. node0
          11 kB
          Pavel Drobek
        2. node1
          13 kB
          Pavel Drobek
        3. operator.log
          43 kB
          Pavel Drobek

        Issue Links

          Activity

            People

              remerson@redhat.com Ryan Emerson
              pdrobek@redhat.com Pavel Drobek
              Pavel Drobek Pavel Drobek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: