Details
-
Bug
-
Resolution: Done
-
Major
-
RHDG 8.3 GA
-
None
-
None
-
False
-
None
-
False
Description
Having enabled the compactor to remove content from persistent storage I have encounter an exception during testing.
server.log
2022-03-07 14:38:21,993 WARN (blocking-thread--p3-t10) [org.infinispan.persistence.sifs.Compactor] Compactor encountered an exception java.lang.NullPointerException at org.infinispan.persistence.sifs.Compactor.compactSingleFile(Compactor.java:371) at org.infinispan.persistence.sifs.Compactor.accept(Compactor.java:273) at io.reactivex.rxjava3.internal.subscribers.LambdaSubscriber.onNext(LambdaSubscriber.java:65) at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:481) at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.tryEmit(FlowableFlatMap.java:302) at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onNext(FlowableFlatMap.java:628) at io.reactivex.rxjava3.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FlowableSwitchIfEmpty.java:59) at io.reactivex.rxjava3.internal.subscriptions.ScalarSubscription.request(ScalarSubscription.java:55) at io.reactivex.rxjava3.internal.subscriptions.SubscriptionArbiter.setSubscription(SubscriptionArbiter.java:99) at io.reactivex.rxjava3.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onSubscribe(FlowableSwitchIfEmpty.java:51) at io.reactivex.rxjava3.internal.operators.flowable.FlowableJust.subscribeActual(FlowableJust.java:34) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15696) at io.reactivex.rxjava3.internal.operators.flowable.FlowableSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FlowableSwitchIfEmpty.java:71) at io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onComplete(BasicFuseableSubscriber.java:120) at io.reactivex.rxjava3.internal.operators.flowable.FlowableTake$TakeSubscriber.onComplete(FlowableTake.java:96) at io.reactivex.rxjava3.internal.subscriptions.EmptySubscription.complete(EmptySubscription.java:69) at io.reactivex.rxjava3.internal.operators.flowable.FlowableEmpty.subscribeActual(FlowableEmpty.java:34) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15696) at io.reactivex.rxjava3.internal.operators.flowable.FlowableTakePublisher.subscribeActual(FlowableTakePublisher.java:38) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750) at io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750) at io.reactivex.rxjava3.internal.operators.flowable.FlowableSwitchIfEmpty.subscribeActual(FlowableSwitchIfEmpty.java:32) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15750) at io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15696) at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.onNext(FlowableFlatMap.java:162) at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$ObserveOnSubscriber.runAsync(FlowableObserveOn.java:402) at io.reactivex.rxjava3.internal.operators.flowable.FlowableObserveOn$BaseObserveOnSubscriber.run(FlowableObserveOn.java:176) at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker$BooleanRunnable.run(ExecutorScheduler.java:322) at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.runEager(ExecutorScheduler.java:287) at io.reactivex.rxjava3.internal.schedulers.ExecutorScheduler$ExecutorWorker.run(ExecutorScheduler.java:248) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:829)
I can reproduce using the following simple config (I first encountered this using a far more complex config but this will do to reproduce)
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
urn:infinispan:server:13.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd
urn:org:jgroups http://www.jgroups.org/schema/jgroups-4.0.xsd"
xmlns="urn:infinispan:config:13.0"
xmlns:server="urn:infinispan:server:13.0"> <server xmlns="urn:infinispan:server:13.0">
<interfaces>
<interface name="public">
<inet-address value="<your IP here>"/>
</interface>
</interfaces> <socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
<socket-binding name="default" port="11222"/>
</socket-bindings> <security>
<security-realms>
<security-realm name="default"/>
</security-realms>
</security> <endpoints socket-binding="default" security-realm="default"/>
</server> <cache-container name="test-container" default-cache="testcache" statistics="false">
<local-cache name="testcache" statistics="true">
<encoding>
<key media-type="text/plain; charset=UTF-8"/>
<value media-type="text/plain; charset=UTF-8"/>
</encoding>
<memory max-size="2GB" when-full="REMOVE" />
<expiration lifespan="60000" />
<persistence passivation="false">
<file-store shared="false" path="data" fetch-state="true" preload="false" purge="false">
<data path="data"/>
<index path="index"/>
</file-store>
</persistence>
<locking concurrency-level="10" acquire-timeout="30000"/>
</local-cache>
</cache-container>
</infinispan>
The system is Red Hat Enterprise Linux 7.9 using Java 11 provided by Red Hat
$ java -version openjdk version "11.0.13" 2021-10-19 LTS OpenJDK Runtime Environment 18.9 (build 11.0.13+8-LTS) OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8-LTS, mixed mode, sharing)
Java options to start infinispan
/usr/lib/jvm/jre-11/bin/java -server -Xms12288m -Xmx12288m -XX:+UseG1GC -XX:MetaspaceSize=32M -XX:MaxMetaspaceSize=64m -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Dinfinispan.server.home.path=/opt/infinisp an/latest -Dinfinispan.server.log.path=/var/log/infinispan -Dio.netty.native.workdir=/opt/infinispan/tmp -classpath :/opt/infinispan/latest/boot/infinispan-server-runtime-13.0.6.Final-loader.jar org.infinispan.server.loader.Loader org.inf inispan.server.Bootstrap
Simple code that can reproduce the issue (within 15 to 20 minutes)
[^infinispan-test.zip]
I typically execute it with 10000 in the first param but it has never made it to 3000 before Infinispan throws the exception in the server.log
I have configured logging for the compactor thread to TRACE to see when its running. Once it throws the exception it never runs again. At this point, you can restart Infinispan and it does start the compactor again and can even remove more content but seems to fail on the same file ID again (very quickly) throwing the same exception.
I've written the code (and above config) as a minimal reproducible example. The goal of the real code will be to stream some large content into an Infinispan cluster of 3 using a replicated (or perhaps distributed) cache for retrieval at a later time. Real world expiry will be in days rather than minutes (tested with days TTL and can still reproduce).
Is there something wrong/missing with/from the config that is causing this exception or perhaps something the code should be doing ? Is there a different way to reduce the used size of the content on the disk without using the compactor that I could use instead? Once the compactor stops working the disk will eventually be consumed and then both the client and infinispan server gets upset.
Attachments
Issue Links
- clones
-
-
- Closed
-
