Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-2018

[JDG65] Switch to standalone.sh breaks down cluster scaling

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: JDG65 1.5.0.GA
    • Fix Version/s: JDG65 1.5.0.GA
    • Component/s: JDG6
    • Labels:
      None

      Description

      Standalone.sh is meant for running single datagrid instance. Running datagrid with this script breaks down scaling during which servers throws OutdatedTopologyExceptions and new nodes fail to start cache services (see below). Also clustered.sh loaded config from clustered.conf file which set numerous envs, mainly JAVA_OPTS and now this file is ignored.

      15:03:47,606 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-1) ISPN000136: Execution error: org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 4, got 5
      	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:264) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:88) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:454) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:188) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:123) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:182) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:168) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      
      15:05:55,290 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1936) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:179)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.CacheImpl.start(CacheImpl.java:823)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:595)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:550)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:421)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:435)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:85)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on jdg-cache-1-rw7qd/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:221)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_141]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_141]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141]
      	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:176)
      ... 23 more
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  bdecoste William Decoste
                  Reporter:
                  pdrobek Pavel Drobek
                  Tester:
                  Pavel Drobek
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: