Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-8416

Error executing MassIndexer using off-heap

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 9.2.0.Alpha2
    • 9.2.0.Alpha1
    • None
    • None

    Description

      Invoking a re-index operation on startup from the API

      SearchManager searchResourceManager = Search.getSearchManager(resource);
CompletableFuture<Void> reindexResource = searchResourceManager.getMassIndexer().startAsync();
SearchManager searchResourceTypeManager = Search.getSearchManager(resourceType);
CompletableFuture<Void> reindexResourceType = searchResourceTypeManager.getMassIndexer().startAsync();
CompletableFuture.allOf(reindexResource, reindexResourceType).get();

      Throws the following Exception

      14:01:23,537 ERROR [org.hawkular.inventory.service.InventoryConfig] (ServerService Thread Pool -- 62) HAWKINV100005: Error reindexing caches: java.util.concurrent.ExecutionException: org.infinispan.commons.CacheException: ISPN014018: Error executing MassIndexer
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
	at org.hawkular.inventory.service.InventoryConfig.init(InventoryConfig.java:111)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptor.java:96)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doLifecycleInterception(Jsr299BindingsInterceptor.java:114)
	at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocation(Jsr299BindingsInterceptor.java:103)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:437)
	at org.jboss.weld.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:73)
	at org.jboss.as.weld.ejb.EjbRequestScopeActivationInterceptor.processInvocation(EjbRequestScopeActivationInterceptor.java:83)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.weld.injection.WeldInjectionInterceptor.processInvocation(WeldInjectionInterceptor.java:53)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.component.AroundConstructInterceptorFactory$1.processInvocation(AroundConstructInterceptorFactory.java:28)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.weld.injection.WeldInterceptorInjectionInterceptor.processInvocation(WeldInterceptorInjectionInterceptor.java:56)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.weld.ejb.Jsr299BindingsCreateInterceptor.processInvocation(Jsr299BindingsCreateInterceptor.java:100)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:275)
	at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
	at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.as.ejb3.component.singleton.StartupCountDownInterceptor.processInvocation(StartupCountDownInterceptor.java:25)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
	at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:124)
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:138)
	at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: org.infinispan.commons.CacheException: ISPN014018: Error executing MassIndexer
	at org.infinispan.query.impl.massindex.DistributedExecutorMassIndexer.lambda$null$0(DistributedExecutorMassIndexer.java:73)
	at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
	at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
	at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart.lambda$execute$1(DefaultExecutorService.java:1086)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Indexing only works with entries keyed on Strings, primitives and classes that have the @Transformable annotation - you passed in a class org.infinispan.commons.marshall.WrappedByteArray. Alternatively, see org.infinispan.query.SearchManager#registerKeyTransformer
	at org.infinispan.query.backend.KeyTransformationHandler.keyToString(KeyTransformationHandler.java:154)
	at org.infinispan.query.impl.massindex.IndexUpdater.updateIndex(IndexUpdater.java:67)
	at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:101)
	at org.infinispan.query.impl.massindex.IndexWorker.call(IndexWorker.java:34)
	at org.infinispan.commands.read.DistributedExecuteCommand.invokeAsync(DistributedExecuteCommand.java:99)
	at org.infinispan.distexec.DefaultExecutorService$LocalDistributedTaskPart.lambda$execute$1(DefaultExecutorService.java:1074)
	... 3 more

      Infinispan 9.2.0.Alpha1 is running in library mode embedded in a .war application with the following configuration:

        <cache-container name="hawkular-inventory">
    <jmx duplicate-domains="true" />
    <local-cache name="resource" statistics="true">
      <transaction mode="NON_XA"/>
      <persistence>
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" modification-queue-size="10000" />
        </file-store>
      </persistence>
      <memory>
        <!--
          <binary eviction="COUNT" size="10000" />
        -->
        <!--
          <object size="5000" />
        -->
        <off-heap eviction="COUNT" size="100000" />
        <!--
          <off-heap eviction="COUNT" size="100000" />
        -->
      </memory>
      <indexing index="LOCAL">
        <indexed-entities>
          <indexed-entity>org.hawkular.inventory.service.ispn.IspnResource</indexed-entity>
        </indexed-entities>
        <property name="default.indexmanager">near-real-time</property>
        <property name="default.directory_provider">infinispan</property>
        <property name="default.chunk_size">128000</property>
        <property name="default.locking_cachename">resource_indexes_locking</property>
        <property name="default.data_cachename">resource_indexes_data</property>
        <property name="default.metadata_cachename">resource_indexes_metadata</property>
        <!-- The default is 10, but we don't want to waste many cycles in merging
         (tune for writes at cost of reader fragmentation) -->

        <property name="default.indexwriter.merge_factor">30</property>
        <!-- Never create segments larger than 1GB -->
        <property name="default.indexwriter.merge_max_size">1024</property>
        <!-- IndexWriter flush buffer size in MB -->
        <property name="default.indexwriter.ram_buffer_size">64</property>
        <!-- Enable sharding on writers -->
        <property name="default.sharding_strategy.nbr_of_shards">6</property>
        <property name="lucene_version">LUCENE_CURRENT</property>
      </indexing>
    </local-cache>
    <local-cache name="resource_indexes_metadata">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE"/>
    </local-cache>
    <local-cache name="resource_indexes_data">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE" />
    </local-cache>
    <local-cache name="resource_indexes_locking">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE" />
    </local-cache>

    <local-cache name="resource_type" statistics="true">
      <transaction mode="NON_XA"/>
      <persistence>
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" modification-queue-size="10000" />
        </file-store>
      </persistence>
      <indexing index="LOCAL">
        <indexed-entities>
          <indexed-entity>org.hawkular.inventory.service.ispn.IspnResourceType</indexed-entity>
        </indexed-entities>
        <property name="default.indexmanager">near-real-time</property>
        <property name="default.directory_provider">infinispan</property>
        <property name="default.chunk_size">128000</property>
        <property name="default.locking_cachename">resource_type_indexes_locking</property>
        <property name="default.data_cachename">resource_type_indexes_data</property>
        <property name="default.metadata_cachename">resource_type_indexes_metadata</property>
        <!-- The default is 10, but we don't want to waste many cycles in merging
         (tune for writes at cost of reader fragmentation) -->

        <property name="default.indexwriter.merge_factor">30</property>
        <!-- Never create segments larger than 1GB -->
        <property name="default.indexwriter.merge_max_size">1024</property>
        <!-- IndexWriter flush buffer size in MB -->
        <property name="default.indexwriter.ram_buffer_size">64</property>
        <!-- Enable sharding on writers -->
        <property name="default.sharding_strategy.nbr_of_shards">6</property>
        <property name="lucene_version">LUCENE_CURRENT</property>
      </indexing>
    </local-cache>
    <local-cache name="resource_type_indexes_metadata">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE"/>
    </local-cache>
    <local-cache name="resource_type_indexes_data">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE" />
    </local-cache>
    <local-cache name="resource_type_indexes_locking">
      <persistence passivation="false">
        <file-store preload="true" fetch-state="true" read-only="false" purge="false" path="${jboss.server.data.dir}/hawkular-inventory">
          <write-behind thread-pool-size="5" />
        </file-store>
      </persistence>
      <indexing index="NONE" />
    </local-cache>
  </cache-container>

      At initialization, it tries to re-index existing resources*.dat files for data and lucene indexes.

      Attachments

        Issue Links

          causes

          Bug - A problem that impairs or prevents the functions of the product. JDG-1360 Error executing MassIndexer when using off-heap

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              gfernand@redhat.com Gustavo Fernandes (Inactive)
              lponce@redhat.com Lucas Ponce
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: