Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9527

Initial state transfer timed out for cache org.infinispan.CONFIG

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 9.4.0.Final
    • 9.3.0.Final
    • Cloud
    • Hide

      I assume this is race condition. I can not provide reliable steps to reproduce because issue randomly pop ups.

      Show
      I assume this is race condition. I can not provide reliable steps to reproduce because issue randomly pop ups.

    Description

      During starting of caches, several nodes fails due following exception (IPv4):

      Cloud access error! org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
org.infinispan.manager.EmbeddedCacheManagerStartupException: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:271) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	...
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart() on object of type GlobalConfigurationManagerImpl
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	... 33 more
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:83) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	... 33 more
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache org.infinispan.CONFIG on 0125f36a-2860-4107-810a-d087224c9105-39521
	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:233) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.security.Security.doPrivileged(Security.java:49) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:689) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:607) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:244) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1051) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:421) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:644) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:589) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:475) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:461) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:447) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.getStateCache(GlobalConfigurationManagerImpl.java:97) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.globalstate.impl.GlobalConfigurationManagerImpl.postStart(GlobalConfigurationManagerImpl.java:80) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_131]
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:1.8.0_131]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:1.8.0_131]
	at org.infinispan.commons.util.SecurityActions.lambda$invokeAccessibly$0(SecurityActions.java:79) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.doPrivileged(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.SecurityActions.invokeAccessibly(SecurityActions.java:76) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:968) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.lambda$invokePrioritizedMethods$6(AbstractComponentRegistry.java:703) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.lambda$run$1(SecurityActions.java:72) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.security.Security.doPrivileged(Security.java:44) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.SecurityActions.run(SecurityActions.java:71) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:696) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.AbstractComponentRegistry.postStart(AbstractComponentRegistry.java:620) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	at org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:267) ~[infinispan-embedded-9.3.0.Final.jar:9.3.0.Final]
	... 33 more

      Attachments

        1. cluster_config_distr.xml
          15 kB
        2. image.png
          image.png
          3 kB
        3. infinispan.log
          39 kB
        4. jgroupsObjectModel.xml
          2 kB
        5. master.zip
          1.22 MB

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. JGRP-2294 ForkProtocolStack ignores UnknownForkHandler for message batches

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Resolved
          is related to

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. ISPN-9546 Upgrade to JGroups 4.0.15.Final

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              dberinde@redhat.com Dan Berindei (Inactive)
              stehlik.michal Michal Stehlik (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: