2021-05-06 15:35:59,917 WARN [io.qua.qut.run.ContentTypes] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream 2021-05-06 15:36:00,125 INFO [io.quarkus] (main) config-generator 2.1.4.CD20210504-temporary-redhat-00001 on JVM (powered by Quarkus 1.7.6.Final-redhat-00015) started in 1.795s. 2021-05-06 15:36:00,125 INFO [io.quarkus] (main) Profile prod activated. 2021-05-06 15:36:00,125 INFO [io.quarkus] (main) Installed features: [cdi, qute] 2021-05-06 15:36:01,520 INFO [io.quarkus] (main) config-generator stopped in 0.002s 15:36:05,416 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.11+9-LTS 15:36:05,424 INFO (main) [BOOT] JVM arguments = [-server, -Xmx256m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=redhat-datagrid-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.1.2.CD20210504-temporary-redhat-00001-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap] 15:36:05,425 INFO (main) [BOOT] PID = 97 15:36:05,614 INFO (main) [org.infinispan.SERVER] ISPN080000: Red Hat Data Grid Server starting 15:36:05,615 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: infinispan.xml 15:36:05,615 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml 15:36:08,723 INFO (main) [org.wildfly.openssl.SSL] WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1g FIPS 21 Apr 2020 15:36:08,724 INFO (main) [org.infinispan.SECURITY] ISPN000946: Using OpenSSL Provider 15:36:09,417 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 15:36:09,417 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 15:36:09,419 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory' 15:36:09,420 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory' 15:36:10,820 WARN (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal 15:36:10,922 INFO (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller' 15:36:13,618 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.TimestampSchemaImpl 15:36:13,618 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.EmptySchemaImpl 15:36:13,619 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.java.CommonTypesSchema 15:36:13,619 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.DurationSchemaImpl 15:36:13,619 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.AnySchemaImpl 15:36:13,620 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.WrappersSchemaImpl 15:36:13,620 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.java.CommonContainerTypesSchema 15:36:13,621 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.query.core.stats.impl.PersistenceContextInitializerImpl 15:36:13,622 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Red Hat Data Grid 'Infinispan 12.1.2.CD20210504-temporary-redhat-00001' 8.2.0.GA 15:36:14,224 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel data-grid-source with stack image-tcp 15:36:14,422 WARN (main) [org.jgroups.stack.Configurator] JGRP000014: STABLE.stability_delay has been deprecated: always 0 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.infinispan.commons.util.Util (file:/opt/infinispan/lib/infinispan-commons-12.1.2.CD20210504-temporary-redhat-00001.jar) to constructor com.sun.jndi.dns.DnsContextFactory() WARNING: Please consider reporting this to the maintainers of org.infinispan.commons.util.Util WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 15:36:16,742 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [data-grid-source-0-25830|0] (1) [data-grid-source-0-25830] 15:36:16,752 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel data-grid-source local address is data-grid-source-0-25830, physical addresses are [10.131.1.214:7800] 15:36:16,914 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=12.1.2.CD20210504-temporary-redhat-00001 timestamp=2021-05-06T15:36:16.913100Z 15:36:17,833 INFO (main) [org.jboss.threads] JBoss Threads version 2.3.3.Final-redhat-00001 15:36:18,030 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 15:36:19,720 INFO (main) [org.infinispan.server.core.RequestTracer] OpenTracing integration is disabled 15:36:20,122 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Started connector HotRod (internal) 15:36:20,536 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 15:36:21,213 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SINGLE_PORT (default) listening on 10.131.1.214:11222 15:36:21,213 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'data-grid-source-0-25830' listening on https://10.131.1.214:11222 15:36:21,315 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 15:36:21,317 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SINGLE_PORT (admin) listening on 10.131.1.214:11223 15:36:21,317 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'data-grid-source-0-25830' listening on http://10.131.1.214:11223 15:36:21,418 INFO (main) [org.infinispan.SERVER] ISPN080001: Red Hat Data Grid Server 8.2.0.GA started in 15802ms 15:37:08,622 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [data-grid-source-0-25830|1] (2) [data-grid-source-0-25830, data-grid-source-1-23351] 15:37:08,627 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] ISPN100000: Node data-grid-source-1-23351 joined the cluster 15:37:10,776 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,079 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,087 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,091 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:11,282 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,382 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,384 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,388 INFO (non-blocking-thread--p2-t2) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,471 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:11,478 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,482 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,486 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:11,494 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,579 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,583 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,586 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:11,674 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,688 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,770 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,774 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:11,785 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:11,871 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:11,875 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:11,879 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:12,876 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:13,073 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:13,077 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:13,081 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:37:26,788 WARN (blocking-thread--p3-t2) [org.infinispan.encoding.impl.StorageConfigurationManager] ISPN000599: Configuration for cache 'someCache' does not define the encoding for keys or values. If you use operations that require data conversion or queries, you should configure the cache with a specific MediaType for keys or values. 15:37:26,978 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], phase READ_OLD_WRITE_ALL, topology id 2 15:37:27,071 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3 15:37:27,075 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4 15:37:27,079 INFO (jgroups-7,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351], topology id 5 15:38:15,523 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [data-grid-source-0-25830|2] (3) [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869] 15:38:15,523 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] ISPN100000: Node backup-6869 joined the cluster 15:38:16,498 ERROR (blocking-thread--p3-t1) [org.infinispan.stats.impl.ClusterCacheStatsImpl] Could not execute cluster wide cache stats operation java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412) at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:118) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:117) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:210) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsLong(AbstractClusterStats.java:201) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getInvalidations(ClusterCacheStatsImpl.java:379) at org.infinispan.factories.impl.MBeanMetadata$AttributeMetadata.lambda$getter$0(MBeanMetadata.java:137) at org.infinispan.metrics.impl.MetricsCollector.lambda$registerMetrics$0(MetricsCollector.java:139) at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492) at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192) at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158) at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73) at org.infinispan.rest.resources.MetricsResource.lambda$metrics$1(MetricsResource.java:80) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.infinispan.commons.CacheException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at org.infinispan.stats.impl.ClusterCacheStatsImpl.lambda$updateStats$0(ClusterCacheStatsImpl.java:109) at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$6(AllClusterExecutor.java:192) at org.infinispan.manager.impl.AbstractClusterExecutor.consumeResponse(AbstractClusterExecutor.java:65) at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$7(AllClusterExecutor.java:191) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:46) at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1415) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1317) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:133) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1458) at org.jgroups.JChannel.up(JChannel.java:784) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913) at org.jgroups.protocols.FRAG3.up(FRAG3.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ... 1 more Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:177) at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1407) ... 25 more 15:38:16,500 WARN (blocking-thread--p3-t1) [io.smallrye.metrics] Unable to export metric cache_manager_default_cache_someCache_cluster_cache_stats_invalidations org.infinispan.commons.CacheException: Could not execute cluster wide cache stats operation at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:120) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:210) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsLong(AbstractClusterStats.java:201) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getInvalidations(ClusterCacheStatsImpl.java:379) at org.infinispan.factories.impl.MBeanMetadata$AttributeMetadata.lambda$getter$0(MBeanMetadata.java:137) at org.infinispan.metrics.impl.MetricsCollector.lambda$registerMetrics$0(MetricsCollector.java:139) at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492) at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192) at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158) at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73) at org.infinispan.rest.resources.MetricsResource.lambda$metrics$1(MetricsResource.java:80) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at java.base/java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:412) at java.base/java.util.concurrent.CompletableFuture.join(CompletableFuture.java:2044) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:118) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:117) ... 18 more Caused by: org.infinispan.commons.CacheException: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at org.infinispan.stats.impl.ClusterCacheStatsImpl.lambda$updateStats$0(ClusterCacheStatsImpl.java:109) at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$6(AllClusterExecutor.java:192) at org.infinispan.manager.impl.AbstractClusterExecutor.consumeResponse(AbstractClusterExecutor.java:65) at org.infinispan.manager.impl.AllClusterExecutor.lambda$submitConsumer$7(AllClusterExecutor.java:191) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.infinispan.remoting.transport.AbstractRequest.complete(AbstractRequest.java:67) at org.infinispan.remoting.transport.impl.SingleTargetRequest.onResponse(SingleTargetRequest.java:46) at org.infinispan.remoting.transport.impl.RequestRepository.addResponse(RequestRepository.java:51) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1415) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1317) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:133) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1458) at org.jgroups.JChannel.up(JChannel.java:784) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913) at org.jgroups.protocols.FRAG3.up(FRAG3.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.FlowControl.up(FlowControl.java:347) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ... 1 more Caused by: org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:177) at org.infinispan.marshall.exts.ThrowableExternalizer.readObject(ThrowableExternalizer.java:42) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:32) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:49) at org.infinispan.remoting.responses.ExceptionResponse$Externalizer.readObject(ExceptionResponse.java:41) at org.infinispan.marshall.core.GlobalMarshaller.readWithExternalizer(GlobalMarshaller.java:727) at org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:708) at org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:357) at org.infinispan.marshall.core.GlobalMarshaller.objectFromObjectInput(GlobalMarshaller.java:191) at org.infinispan.marshall.core.GlobalMarshaller.objectFromByteBuffer(GlobalMarshaller.java:220) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processResponse(JGroupsTransport.java:1407) ... 25 more 15:38:17,503 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:17,692 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:17,698 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:17,702 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:17,801 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:17,808 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:17,892 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:17,896 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:17,904 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:17,991 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:17,995 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:17,998 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:18,004 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:18,198 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:18,202 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:18,205 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:18,301 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:18,310 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:18,391 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:18,396 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:18,405 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:18,409 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:18,412 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:18,414 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:18,804 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:18,809 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:18,892 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:18,896 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:19,503 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], phase READ_OLD_WRITE_ALL, topology id 6 15:38:19,508 INFO (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7 15:38:19,511 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8 15:38:19,593 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100010: Finished rebalance with members [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869], topology id 9 15:38:29,209 INFO (Thread-1) [org.infinispan.SERVER] ISPN080002: Red Hat Data Grid Server stopping *** Server process (97) received TERM signal *** 15:38:29,219 INFO (ForkJoinPool.commonPool-worker-1) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,233 WARN (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN000312: Lost data because of graceful leaver data-grid-source-1-23351 15:38:29,330 INFO (Thread-1) [org.infinispan.manager.DefaultCacheManager] Stopping cache manager on data-grid-source-0-25830 15:38:29,342 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=someCache]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,380 WARN (jgroups-15,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=someCache] ISPN000312: Lost data because of graceful leaver data-grid-source-1-23351 15:38:29,431 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,483 WARN (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=___protobuf_metadata] ISPN000312: Lost data because of graceful leaver data-grid-source-1-23351 15:38:29,487 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100008: Updating cache members list [data-grid-source-0-25830, backup-6869], topology id 10 15:38:29,512 INFO (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100002: Starting rebalance with members [data-grid-source-0-25830, backup-6869], phase READ_OLD_WRITE_ALL, topology id 11 15:38:29,514 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100008: Updating cache members list [backup-6869], topology id 12 15:38:29,516 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100008: Updating cache members list [backup-6869], topology id 13 15:38:29,517 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,521 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,523 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 10 15:38:29,525 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100011: Entering availability mode AVAILABLE, topology id 10 15:38:29,526 INFO (Thread-1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100008: Updating cache members list [data-grid-source-1-23351, backup-6869], topology id 11 15:38:29,611 WARN (jgroups-5,data-grid-source-0-25830) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER] ISPN000312: Lost data because of graceful leaver data-grid-source-1-23351 15:38:29,613 INFO (Thread-1) [org.infinispan.CLUSTER] ISPN000080: Disconnecting JGroups channel data-grid-source 15:38:29,680 INFO (Thread-1) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=12.1.2.CD20210504-temporary-redhat-00001 timestamp=2021-05-06T15:38:29.680093Z 15:38:29,685 INFO (Thread-1) [org.infinispan.SERVER] ISPN080003: Red Hat Data Grid Server stopped