Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-2018

[JDG65] Switch to standalone.sh breaks down cluster scaling

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • JDG65 1.5.0.GA
    • JDG65 1.5.0.GA
    • JDG6
    • None
    • CLOUD Maintenance Sprint 12

      Standalone.sh is meant for running single datagrid instance. Running datagrid with this script breaks down scaling during which servers throws OutdatedTopologyExceptions and new nodes fail to start cache services (see below). Also clustered.sh loaded config from clustered.conf file which set numerous envs, mainly JAVA_OPTS and now this file is ignored.

      15:03:47,606 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-1) ISPN000136: Execution error: org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 4, got 5
      	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:264) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:88) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:454) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:188) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:123) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:182) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:168) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      
      15:05:55,290 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1936) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:179)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.CacheImpl.start(CacheImpl.java:823)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:595)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:550)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:421)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:435)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:85)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on jdg-cache-1-rw7qd/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:221)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_141]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_141]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141]
      	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:176)
      ... 23 more
      

              wdecoste1@redhat.com William Decoste (Inactive)
              pdrobek@redhat.com Pavel Drobek (Inactive)
              Pavel Drobek Pavel Drobek (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: