Loading...

Type: Bug
Resolution: Obsolete
Priority: Major
Fix Version/s: None
Affects Version/s: 8.1.0.Final
Component/s: None
Labels:
None

Steps to Reproduce:

Hide

I have been unable to reproduce with a single node having multiple CacheManagers. To trigger this, we have a high number of cache.put or cache.remove calls happening across many keys while also handling a high number of cache.get calls. Requests are spread over multiple servers. The cache is backed by a Redis cache store.

Show
I have been unable to reproduce with a single node having multiple CacheManagers. To trigger this, we have a high number of cache.put or cache.remove calls happening across many keys while also handling a high number of cache.get calls. Requests are spread over multiple servers. The cache is backed by a Redis cache store.

During performance tests, we have encountered a small but consistent level of NullPointerExceptions in BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting.

In our system, we have multiple servers acting as a sort of registry. Infinispan is configured with a with a Redis cache store Items are added, removed, and read frequently. We were handling around 8k puts/min, 8k removes per minute, and about 1k gets per minute. We see 0-5 failures per minute on the cache.get calls with the stack below (mostly 0 with sporadic failures).

The relevant code can be seen on 8.2.x branch. The node is null when `node.val` is called.

BoundedEquivalentConcurrentHashMapV8.java

 } else if (evict.state == Recency.HIR_NONRESIDENT) {
    Node<K, V> node = f.find(hash, evict.getKey());
    V prevValue = node.val;
    if (prevValue != NULL_VALUE) {
       node.val = (V) NULL_VALUE;
       map.addCount(-1, -1);
       Node<K, V> nonResidentNode = new Node<K, V>(-1, null, evict.getKey(),
             prevValue, null);
       removedNodes.add(nonResidentNode);
       map.notifyListenerOfRemoval(nonResidentNode, true);
    }
 }

I believe this just needs a node != null check.

Backtrace

2016-10-18 03:37:46,678 i-baa20229 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (EE-ManagedExecutorService-LechmereDefault-Thread-1) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: java.lang.NullPointerException
at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:1532) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3658) [infinispan-commons-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.container.DefaultDataContainer.compute(DefaultDataContainer.java:323) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.persistence.PersistenceUtil.loadAndStoreInDataContainer(PersistenceUtil.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.loadInContext(CacheLoaderInterceptor.java:367) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:362) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.visitDataCommand(CacheLoaderInterceptor.java:183) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:137) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:133) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:123) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataReadCommand(NonTransactionalLockingInterceptor.java:31) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:77) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:107) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:76) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:411) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:403) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:286) [infinispan-core-8.1.0.Final.jar:8.1.0.Final]
at our code

Our cache config (boiled down):

Cache Config

ConfigurationBuilder configBuilder = new ConfigurationBuilder();
configBuilder
    .locking()
        .useLockStriping(false)
        .isolationLevel(IsolationLevel.READ_COMMITTED)
        .concurrencyLevel(12)
    .eviction()
        .strategy(EvictionStrategy.LIRS)
        .type(EvictionType.COUNT).size(10000)
    .clustering()
        .cacheMode(CacheMode.INVALIDATION_SYNC)
    .persistence()
        .passivation(false)
        .addStore(RedisStoreConfigurationBuilder.class)
        .connectionPool()
        .minIdle(6)
        .maxIdle(10)
        .maxTotal(20)
        .minEvictableIdleTime(30000)
        .timeBetweenEvictionRuns(30000)
        .topology(Topology.SERVER)
        .fetchPersistentState(true)
        .database(1)
        .addServer()
        .host("127.0.0.1")
        .port(REDIS_PORT)
        .socketTimeout(5000)
        .connectionTimeout(5000);

is incorporated by

ISPN-6998 Replace bounded hash map with alternative

Closed

Details

Description

Backtrace

Our cache config (boiled down):

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates