Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-2340

AsyncStore.remove() load the bucket from store inducing performance problem

    XMLWordPrintable

Details

    Description

      I'm currently using infinispan-5.1.5 as a local persistent cache (like a persistent map) with no eviction and write-behind for maximum performance.

      I tried the 5.2.0alpha4 version and item processing mean decreased from 4ms to 60 ms.
      So I profiled the application and discovered that the Async.remove(Object) implementation differs from 5.2 to 5.1:

      5.1:

         @Override
         public boolean remove(Object key) {
            enqueue(new Remove(key));
            return true;
         }
      

      5.2:

         @Override
         public boolean remove(Object key) {
            try {
               InternalCacheEntry load = load(key);
               if (load != null) {
                  enqueue(new Remove(key));
                  return true;
               }
               return false;
            } catch (CacheLoaderException e) {
               throw new CacheException("Could not load key/value entries from cacheloader", e);
            }
         }
      

      So each time I remove an entry from my cache, I pay the cost of a bucket load from the store:

      Thread [pool-9-thread-1] (Suspended)	
      	ExtendedRiverUnmarshaller(RiverUnmarshaller).doReadObject(boolean) line: 209	
      	ExtendedRiverUnmarshaller(AbstractObjectInput).readObject() line: 37	
      	JBossMarshaller(AbstractJBossMarshaller).objectFromObjectStream(ObjectInput) line: 163	
      	VersionAwareMarshaller.objectFromObjectStream(ObjectInput) line: 190	
      	CacheMarshaller(AbstractDelegatingMarshaller).objectFromObjectStream(ObjectInput) line: 79	
      	FileCacheStore.objectFromInputStreamInReentrantMode(InputStream) line: 433	
      	FileCacheStore.loadBucket(File) line: 306	
      	FileCacheStore.loadBucket(Integer) line: 280	
      	FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Integer) line: 60	
      	FileCacheStore(BucketBasedCacheStore).loadLockSafe(Object, Object) line: 49	
      	FileCacheStore(LockSupportCacheStore<L>).load(Object) line: 142	
      	AsyncStore(AbstractDelegatingStore).load(Object) line: 120	
      	AsyncStore.remove(Object) line: 163	
      	CacheStoreInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 221	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	CacheLoaderInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	CacheLoaderInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 138	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	EntryWrappingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	EntryWrappingInterceptor.invokeNextAndApplyChanges(InvocationContext, FlagAffectedCommand) line: 211	
      	EntryWrappingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 161	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	NonTransactionalLockingInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	NonTransactionalLockingInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 104	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	NotificationInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	NotificationInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132	
      	NotificationInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	IsMarshallableInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	IsMarshallableInterceptor(CommandInterceptor).handleDefault(InvocationContext, VisitableCommand) line: 132	
      	IsMarshallableInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
      	IsMarshallableInterceptor.visitRemoveCommand(InvocationContext, RemoveCommand) line: 125	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	InvocationContextInterceptor(CommandInterceptor).invokeNextInterceptor(InvocationContext, VisitableCommand) line: 118	
      	InvocationContextInterceptor.handleAll(InvocationContext, VisitableCommand) line: 129	
      	InvocationContextInterceptor.handleDefault(InvocationContext, VisitableCommand) line: 93	
      	InvocationContextInterceptor(AbstractVisitor).visitRemoveCommand(InvocationContext, RemoveCommand) line: 68	
      	RemoveCommand.acceptVisitor(InvocationContext, Visitor) line: 72	
      	InterceptorChain.invoke(InvocationContext, VisitableCommand) line: 347	
      	CacheImpl<K,V>.executeCommandAndCommitIfNeeded(InvocationContext, VisitableCommand) line: 999	
      	CacheImpl<K,V>.remove(Object, EnumSet<Flag>, ClassLoader) line: 290	
      	DecoratedCache<K,V>.remove(Object) line: 324	
      	CacheBidiMap<K,V>(AbstractDualBidiMap<K,V>).remove(Object) line: 199	
      	OrderCache.removePendingByClientId(String) line: 168	
      	OrderEventCacheManager(EventCacheManager).receiveAck(OrderId) line: 166	
      	OrderEventCacheManager(EventCacheManager).cacheEventNotification(IamapiEvent, OrderId) line: 101	
      	OrderMarketFixService.onOrderEvent(OrderEvent) line: 88	
      	OrderEventDispatcher.fireEvent(OrderEvent) line: 52	
      	OrderEventDispatcher.run() line: 67	
      	ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1110	
      	ThreadPoolExecutor$Worker.run() line: 603	
      	Thread.run() line: 722	
      
      

      My config file is:

      	<default>
      		<loaders passivation="false" preload="true" shared="false">
      			<loader class="org.infinispan.loaders.file.FileCacheStore">
      				<properties>
      					<property name="location" value="cache" />
      					<property name="fsyncMode" value="perWrite" />
      				</properties>
      				<async enabled="true" />
      			</loader>
      		</loaders>
      		<expiration reaperEnabled="false" />
      	</default>
      

      (I use perWrite for fsyncMode because we reached linux open files ulimit maximum)

      We cannot consider that as a bug, but this performance problem is a show-stopper for us to upgrade to 5.2.

      Best regards,

      Christophe

      Attachments

        Issue Links

          Activity

            People

              gzamarre Galder ZamarreƱo
              domak Christophe Domas (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: