Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-5259

Disabling rebalancing causes errors when starting the second server in a cluster

    XMLWordPrintable

Details

    Description

      Perform the following steps with Infinispan 7.2 server:

      1. Start server using command: bin/clustered.sh -Djboss.node.name=node0 -Djboss.socket.binding.port-offset=0
      2. Set JMX attribute jboss.infinispan/CacheManager/"clustered"/LocalTopologyManager/rebalancingEnabled to false
      3. Start second serer using command: bin/clustered.sh -Djboss.node.name=node1 -Djboss.socket.binding.port-offset=10

      Result: The first server starts successfully and shows this log message when the JMX attribute is modified:

      09:38:05,620 INFO  [org.infinispan.CLUSTER] (RMI TCP Connection(3)-127.0.0.1) ISPN000309: Rebalancing suspended
      

      The second server starts, but then shows the following error messages:

      =========================================================================
      
        JBoss Bootstrap Environment
      
        JBOSS_HOME: /home/afield/Development/projects/infinispan-public/server/integration/build/target/infinispan-server-7.2.0-SNAPSHOT
      
        JAVA: /usr/lib/jvm/java/bin/java
      
        JAVA_OPTS:  -server -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''
      
      =========================================================================
      
      OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
      09:38:24,626 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final
      09:38:24,895 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.2.Final
      09:38:24,967 INFO  [org.jboss.as] (MSC service thread 1-8) JBAS015899: JBoss Infinispan Server 7.2.0-SNAPSHOT (WildFly 8.1.0.Final) starting
      09:38:26,332 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
      09:38:26,355 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.2.2.Final
      09:38:26,371 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.2.2.Final
      09:38:26,424 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 21) JBAS010280: Activating Infinispan subsystem.
      09:38:26,448 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 22) WFLYIO001: Worker 'default' has auto-configured to 8 core threads with 64 task threads based on your 4 available processors
      09:38:26,464 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 25) JBAS010260: Activating JGroups subsystem.
      09:38:26,476 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 28) JBAS011800: Activating Naming Subsystem
      09:38:26,479 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.3.Final
      09:38:26,493 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 30) JBAS013171: Activating Security Subsystem
      09:38:26,497 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 32) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.
      09:38:26,505 INFO  [org.jboss.as.security] (MSC service thread 1-3) JBAS013170: Current PicketBox version=4.0.21.Beta1
      09:38:26,533 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-8) JBAS010408: Starting JCA Subsystem (IronJacamar 1.1.5.Final)
      09:38:26,577 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) JBAS017502: Undertow 1.0.15.Final starting
      09:38:26,579 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 33) JBAS017502: Undertow 1.0.15.Final starting
      09:38:26,629 INFO  [org.jboss.as.naming] (MSC service thread 1-6) JBAS011802: Starting Naming Service
      09:38:26,889 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-6) JBAS017525: Started server default-server.
      09:38:26,901 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017531: Host default-host starting
      09:38:27,015 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-5) JBAS017519: Undertow HTTP listener default listening on /127.0.0.1:8180
      09:38:27,032 WARN  [org.jgroups.stack.Configurator] (MSC service thread 1-3) JGRP000014: TP.singleton_name has been deprecated: Use fork channels instead
      09:38:27,266 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-8) JBAS015012: Started FileSystemDeploymentService for directory /home/afield/Development/projects/infinispan-public/server/integration/build/target/infinispan-server-7.2.0-SNAPSHOT/standalone/deployments
      09:38:27,632 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) JDGS010000: REST starting
      09:38:28,049 INFO  [org.hibernate.validator.internal.util.Version] (MSC service thread 1-7) HV000001: Hibernate Validator 5.1.0.Final
      09:38:28,075 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered
      09:38:28,111 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket DatagramSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      09:38:28,114 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      09:38:28,117 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the send buffer of socket MulticastSocket was set to 1MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
      09:38:28,119 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-8) JGRP000015: the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
      09:38:28,127 INFO  [org.jboss.resteasy.plugins.validation.AbstractValidatorContextResolver] (MSC service thread 1-7) Unable to find CDI supporting ValidatorFactory. Using default ValidatorFactory
      09:38:28,231 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-7) JBAS017534: Registered web context: /
      09:38:28,232 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-7) JDGS010002: REST mapped to /rest
      09:38:28,285 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [node0/clustered|1] (2) [node0/clustered, node1/clustered]
      09:38:28,395 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is node1/clustered, physical addresses are [127.0.0.1:55300]
      09:38:28,486 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Hoptimus Prime' 7.2.0-SNAPSHOT
      09:38:29,017 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started ___protobuf_metadata cache from clustered container
      09:38:29,046 INFO  [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-6) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
      09:38:29,038 INFO  [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-3) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
      09:38:29,067 INFO  [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-2) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
      09:38:29,073 INFO  [org.infinispan.eviction.impl.EvictionManagerImpl] (MSC service thread 1-8) ISPN000025: wakeUpInterval is <= 0, not starting expired purge thread
      09:39:29,145 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-3) MSC000001: Failed to start service jboss.infinispan.clustered.namedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
      	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache namedCache on node1/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
      	at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	... 23 more
      
      09:39:29,153 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.infinispan.clustered.transactionalCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
      	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactionalCache on node1/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
      	at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	... 23 more
      
      09:39:29,151 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-8) MSC000001: Failed to start service jboss.infinispan.clustered.memcachedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
      	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache memcachedCache on node1/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
      	at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	... 23 more
      
      09:39:29,150 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_31]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_31]
      	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_31]
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:171)
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:423)
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:437)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)
      	at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:116)
      	at org.infinispan.server.infinispan.SecurityActions$4.run(SecurityActions.java:113)
      	at org.infinispan.security.Security.doPrivileged(Security.java:76)
      	at org.infinispan.server.infinispan.SecurityActions.doPrivileged(SecurityActions.java:60)
      	at org.infinispan.server.infinispan.SecurityActions.startCache(SecurityActions.java:121)
      	at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:79)
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881) [jboss-msc-1.2.2.Final.jar:1.2.2.Final]
      	... 3 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1/clustered
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:219)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_31]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_31]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_31]
      	at java.lang.reflect.Method.invoke(Method.java:483) [rt.jar:1.8.0_31]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      	... 23 more
      
      09:39:29,166 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
          ("subsystem" => "infinispan"),
          ("cache-container" => "clustered"),
          ("distributed-cache" => "default")
      ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.default" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache default on node1/clustered"}}
      09:39:29,169 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
          ("subsystem" => "infinispan"),
          ("cache-container" => "clustered"),
          ("distributed-cache" => "memcachedCache")
      ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.memcachedCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache memcachedCache on node1/clustered"}}
      09:39:29,172 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
          ("subsystem" => "infinispan"),
          ("cache-container" => "clustered"),
          ("distributed-cache" => "namedCache")
      ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.namedCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache namedCache on node1/clustered"}}
      09:39:29,174 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014613: Operation ("add") failed - address: ([
          ("subsystem" => "infinispan"),
          ("cache-container" => "clustered"),
          ("distributed-cache" => "transactionalCache")
      ]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.clustered.transactionalCache" => "org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
          Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
          Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache transactionalCache on node1/clustered"}}
      09:39:29,211 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
      JBAS014777:   Services which failed to start:      service jboss.infinispan.clustered.transactionalCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.transactionalCache: Failed to start service
            service jboss.infinispan.clustered.default: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.default: Failed to start service
            service jboss.infinispan.clustered.namedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.namedCache: Failed to start service
            service jboss.infinispan.clustered.memcachedCache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.memcachedCache: Failed to start service
      
      09:39:29,342 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10090/management
      09:39:29,343 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10090
      09:39:29,343 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss Infinispan Server 7.2.0-SNAPSHOT (WildFly 8.1.0.Final) started (with errors) in 65043ms - Started 147 of 175 services (7 services failed or missing dependencies, 64 services are lazy, passive or on-demand)
      

      Attachments

        Activity

          People

            pruivo@redhat.com Pedro Ruivo
            rhn-support-afield Alan Field
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: