Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-2018

[JDG65] Switch to standalone.sh breaks down cluster scaling

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • JDG65 1.5.0.GA
    • JDG65 1.5.0.GA
    • JDG6
    • None
    • CLOUD Maintenance Sprint 12

    Description

      Standalone.sh is meant for running single datagrid instance. Running datagrid with this script breaks down scaling during which servers throws OutdatedTopologyExceptions and new nodes fail to start cache services (see below). Also clustered.sh loaded config from clustered.conf file which set numerous envs, mainly JAVA_OPTS and now this file is ignored.

      15:03:47,606 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (remote-thread-1) ISPN000136: Execution error: org.infinispan.statetransfer.OutdatedTopologyException: Cache topology changed while the command was executing: expected 4, got 5
      	at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:264) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitPutKeyValueCommand(NonTxDistributionInterceptor.java:88) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:366) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:454) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:188) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:46) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:330) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:268) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:123) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:182) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:168) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:43) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:92) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:47) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:185) [infinispan-core-6.3.1.Final-redhat-1.jar:6.3.1.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      
      15:05:55,290 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-5) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1936) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_141]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_141]
      	at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_141]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:179)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.CacheImpl.start(CacheImpl.java:823)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:595)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:550)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:421)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:435)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:104)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:101)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:48)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:109)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:85)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1980) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1913) [jboss-msc-1.1.5.Final-redhat-1.jar:1.1.5.Final-redhat-1]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on jdg-cache-1-rw7qd/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:221)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_141]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_141]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_141]
      	at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_141]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:176)
      ... 23 more
      

      Attachments

        Issue Links

          Activity

            People

              wdecoste1@redhat.com William Decoste (Inactive)
              pdrobek@redhat.com Pavel Drobek
              Pavel Drobek Pavel Drobek
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: