Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-12224

Cluster in a confusing state after restarted from graceful shutdown - no hint for waiting on complete restarted

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 14.0.8.Final, 15.0.0.Final
    • 11.0.3.Final, 12.0.0.Final
    • Core
    • None
    • Hide

      Start a cluster of (4) nodes
      add entries

      use CLI "shutdown cluster"

      start only one node and try to read entries

      The following log will be shown:

      14:17:45,611 ERROR (blocking-thread-testP1-p3-t2) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PrepareCommand on Cache '___protobuf_metadata', writing keys [message.proto, .errors, message.proto.errors] java.lang.IllegalArgumentException: Command does not have a topology id
 at org.infinispan.remoting.rpc.RpcManagerImpl.checkTopologyId(RpcManagerImpl.java:163)
 at org.infinispan.remoting.rpc.RpcManagerImpl.toCacheRpcCommand(RpcManagerImpl.java:327)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommandOnAll(RpcManagerImpl.java:206)
 at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:277)
 at org.infinispan.interceptors.distribution.TxDistributionInterceptor.lambda$visitPrepareCommand$5(TxDistributionInterceptor.java:261)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:86)
 at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:249)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:79)
 at org.infinispan.interceptors.impl.CacheWriterInterceptor.visitPrepareCommand(CacheWriterInterceptor.java:112)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
 at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:103)
 at org.infinispan.query.remote.impl.ProtobufMetadataManagerInterceptor.visitPrepareCommand(ProtobufMetadataManagerInterceptor.java:231)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:79)
 at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:190)
 at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:912)
 at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:185)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:103)
 at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:122)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:153)
 at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:124)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190)
 at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:203)
 at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
 at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128)
 at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61)
 at org.infinispan.interceptors.impl.BatchingInterceptor.handleDefault(BatchingInterceptor.java:69)
 at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59)
 at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53)
 at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131)
 at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208)
 at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49)
 at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:226)
 at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:177)
 at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:905)
 at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:34)
 at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:505)
 at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:331)
 at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:107)
 at org.infinispan.cache.impl.InvocationHelper.lambda$executeCommandAsyncWithInjectedTx$1(InvocationHelper.java:237)
 at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
 at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
 at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
 at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
 at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
 at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
 at java.lang.Thread.run(Thread.java:748)
      Show
      Start a cluster of (4) nodes add entries use CLI "shutdown cluster" start only one node and try to read entries The following log will be shown: 14:17:45,611 ERROR (blocking-thread-testP1-p3-t2) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PrepareCommand on Cache '___protobuf_metadata', writing keys [message.proto, .errors, message.proto.errors] java.lang.IllegalArgumentException: Command does not have a topology id at org.infinispan.remoting.rpc.RpcManagerImpl.checkTopologyId(RpcManagerImpl.java:163) at org.infinispan.remoting.rpc.RpcManagerImpl.toCacheRpcCommand(RpcManagerImpl.java:327) at org.infinispan.remoting.rpc.RpcManagerImpl.invokeCommandOnAll(RpcManagerImpl.java:206) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.prepareOnAffectedNodes(TxDistributionInterceptor.java:277) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.lambda$visitPrepareCommand$5(TxDistributionInterceptor.java:261) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:86) at org.infinispan.interceptors.distribution.TxDistributionInterceptor.visitPrepareCommand(TxDistributionInterceptor.java:249) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:79) at org.infinispan.interceptors.impl.CacheWriterInterceptor.visitPrepareCommand(CacheWriterInterceptor.java:112) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:103) at org.infinispan.query.remote.impl.ProtobufMetadataManagerInterceptor.visitPrepareCommand(ProtobufMetadataManagerInterceptor.java:231) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenApply(BaseAsyncInterceptor.java:79) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:190) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:912) at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:185) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:103) at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:122) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:153) at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:124) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitCommand(TransactionSynchronizerInterceptor.java:41) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:190) at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:203) at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:69) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) at org.infinispan.interceptors.impl.BatchingInterceptor.handleDefault(BatchingInterceptor.java:69) at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:53) at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:131) at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:208) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:49) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invokeAsync(AsyncInterceptorChainImpl.java:226) at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:177) at org.infinispan.transaction.impl.TransactionTable.afterCompletion(TransactionTable.java:905) at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:34) at org.infinispan.commons.tx.TransactionImpl.notifyAfterCompletion(TransactionImpl.java:505) at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:331) at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:107) at org.infinispan.cache.impl.InvocationHelper.lambda$executeCommandAsyncWithInjectedTx$1(InvocationHelper.java:237) at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.lang.Thread.run(Thread.java:748)
    • Documentation (Ref Guide, User Guide, etc.), User Experience

      After a cluster is stopped with "shutdown cluster" and incomplete restart there is no WARN or INFO message that the cluster is in an incomplete state if not all nodes are back.

      If there is a single node started it is still possible to add new entries!!
      As well as entries can be read.
      But the server will throw Exceptions.

      The expectation is to have log messages with a statement that the cluster of (a,b, ...)  is incomplete started after graceful shutdown and the missing nodes are (x,y,...)

      It should not be possible to access caches.

      There should be a CLI/JMX option to interrupt the graceful start and set the cluster to a working state - even if there is a possible loss of data in this case.
       
       

            rh-ee-jbolina Jose Bolina
            rhn-support-wfink Wolf Fink
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: