Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9527

Initial state transfer timed out for cache org.infinispan.CONFIG

    Details

    • Steps to Reproduce:
      Hide

      I assume this is race condition. I can not provide reliable steps to reproduce because issue randomly pop ups.

      Show
      I assume this is race condition. I can not provide reliable steps to reproduce because issue randomly pop ups.

      Description

      During starting of caches, several nodes fails due following exception (IPv4):

      Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
      org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	...
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
      	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	... 33 more
      Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
      	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	... 33 more
      Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
      	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
      	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
      	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
      	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
      	... 33 more
      

        Gliffy Diagrams

          Attachments

          1. cluster_config_distr.xml
            15 kB
          2. image.png
            image.png
            3 kB
          3. infinispan.log
            39 kB
          4. jgroupsObjectModel.xml
            2 kB
          5. master.zip
            1.22 MB

            Issue Links

              Activity

                People

                • Assignee:
                  dan.berindei Dan Berindei
                  Reporter:
                  stehlik.michal Michal Stehlik
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: