-
Bug
-
Resolution: Done
-
Major
-
None
-
10.1.8.Final
-
None
-
None
Hello,
We encountered a very strange issue running infinispan with two instances of our application that connect to the same cluster. Once Prometheus is ran along with the other instances, it caused the following error.
Versions:
Karaf: 4.2.6
Infinispan: 10.1.8.Final
Docker: 19.03.12, build 48a66213fe
java.lang.IllegalArgumentException: No marshaller registered for Java type org.infinispan.topology.RebalancingStatus at org.infinispan.protostream.impl.SerializationContextImpl.getMarshallerDelegate(SerializationContextImpl.java:279) ~[422:org.infinispan.protostream:4.3.2.Final] at org.infinispan.protostream.WrappedMessage.writeMessage(WrappedMessage.java:240) ~[422:org.infinispan.protostream:4.3.2.Final] at org.infinispan.protostream.ProtobufUtil.toWrappedStream(ProtobufUtil.java:196) ~[422:org.infinispan.protostream:4.3.2.Final] at org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl.objectToBuffer(PersistenceMarshallerImpl.java:157) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl.objectToByteBuffer(PersistenceMarshallerImpl.java:137) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.persistence.impl.PersistenceMarshallerImpl.objectToByteBuffer(PersistenceMarshallerImpl.java:145) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeRawUnknown(GlobalMarshaller.java:638) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeUnknown(GlobalMarshaller.java:627) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeUnknown(GlobalMarshaller.java:618) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:384) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.BytesObjectOutput.writeObject(BytesObjectOutput.java:26) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:65) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.responses.SuccessfulResponse$Externalizer.writeObject(SuccessfulResponse.java:62) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeInternal(GlobalMarshaller.java:656) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeNonNullableObject(GlobalMarshaller.java:371) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeNullableObject(GlobalMarshaller.java:352) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:181) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.writeObjectOutput(GlobalMarshaller.java:174) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.marshall.core.GlobalMarshaller.objectToBuffer(GlobalMarshaller.java:302) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.sendResponse(JGroupsTransport.java:1355) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$processRequest$8(JGroupsTransport.java:1400) ~[421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ResponseConsumer.accept(GlobalInboundInvocationHandler.java:182) [421:org.infinispan.core:10.1.5.Final] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ResponseConsumer.accept(GlobalInboundInvocationHandler.java:170) [421:org.infinispan.core:10.1.5.Final] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774) [?:?] at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792) [?:?] at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153) [?:?] at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:160) [421:org.infinispan.core:10.1.5.Final] at org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl$RunnableWrapper.run(BlockingTaskAwareExecutorServiceImpl.java:215) [421:org.infinispan.core:10.1.5.Final] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?]
Thanks in advance!
After a deeper research, we figured out that having Prometheus container running along with two instances of our application that connect to one infinispan cluster up at the same time resolved in the error spamming in one of the instances.
To resolve, we had to remove infinispan object being exported via `jmx_prometheus_javaagent`