2021-05-06 15:37:58,796 WARN [io.qua.qut.run.ContentTypes] (main) Unable to detect the content type for templates/jgroups-encrypt; using application/octet-stream 2021-05-06 15:37:58,998 INFO [io.quarkus] (main) config-generator 2.1.4.CD20210504-temporary-redhat-00001 on JVM (powered by Quarkus 1.7.6.Final-redhat-00015) started in 1.800s. 2021-05-06 15:37:58,999 INFO [io.quarkus] (main) Profile prod activated. 2021-05-06 15:37:58,999 INFO [io.quarkus] (main) Installed features: [cdi, qute] 2021-05-06 15:38:00,799 INFO [io.quarkus] (main) config-generator stopped in 0.003s 15:38:04,897 INFO (main) [BOOT] JVM OpenJDK 64-Bit Server VM Red Hat, Inc. 11.0.11+9-LTS 15:38:04,906 INFO (main) [BOOT] JVM arguments = [-server, -Xmx256m, -XX:+ExitOnOutOfMemoryError, -XX:MetaspaceSize=32m, -XX:MaxMetaspaceSize=96m, -Djava.net.preferIPv4Stack=true, -Djava.awt.headless=true, -Dvisualvm.display.name=redhat-datagrid-server, -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager, -Dinfinispan.server.home.path=/opt/infinispan, -classpath, :/opt/infinispan/boot/infinispan-server-runtime-12.1.2.CD20210504-temporary-redhat-00001-loader.jar, org.infinispan.server.loader.Loader, org.infinispan.server.Bootstrap] 15:38:04,906 INFO (main) [BOOT] PID = 96 15:38:05,094 INFO (main) [org.infinispan.SERVER] ISPN080000: Red Hat Data Grid Server starting 15:38:05,094 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: infinispan.xml 15:38:05,095 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml 15:38:08,501 INFO (main) [org.wildfly.openssl.SSL] WFOPENSSL0002 OpenSSL Version OpenSSL 1.1.1g FIPS 21 Apr 2020 15:38:08,503 INFO (main) [org.infinispan.SECURITY] ISPN000946: Using OpenSSL Provider 15:38:09,297 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory' 15:38:09,297 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory' 15:38:09,300 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'iteration-filter-converter-factory' 15:38:09,301 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory' 15:38:10,903 WARN (main) [org.infinispan.PERSISTENCE] ISPN000554: jboss-marshalling is deprecated and planned for removal 15:38:11,005 INFO (main) [org.infinispan.CONTAINER] ISPN000556: Starting user marshaller 'org.infinispan.commons.marshall.ImmutableProtoStreamMarshaller' 15:38:11,402 WARN (main) [org.infinispan.lock.impl.ClusteredLockModuleLifecycle] When the node is configured as a zero-capacity node, you need to specify the number of owners for the lock 15:38:14,213 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.TimestampSchemaImpl 15:38:14,291 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.EmptySchemaImpl 15:38:14,291 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.java.CommonTypesSchema 15:38:14,292 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.DurationSchemaImpl 15:38:14,293 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.AnySchemaImpl 15:38:14,293 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.protobuf.WrappersSchemaImpl 15:38:14,294 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.protostream.types.java.CommonContainerTypesSchema 15:38:14,294 INFO (main) [org.infinispan.query.remote.impl.ProtobufMetadataManagerImpl] ISPN028019: Registering protostream serialization context initializer: org.infinispan.query.core.stats.impl.PersistenceContextInitializerImpl 15:38:14,296 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Red Hat Data Grid 'Infinispan 12.1.2.CD20210504-temporary-redhat-00001' 8.2.0.GA 15:38:14,991 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel data-grid-source with stack image-tcp 15:38:15,194 WARN (main) [org.jgroups.stack.Configurator] JGRP000014: STABLE.stability_delay has been deprecated: always 0 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.infinispan.commons.util.Util (file:/opt/infinispan/lib/infinispan-commons-12.1.2.CD20210504-temporary-redhat-00001.jar) to constructor com.sun.jndi.dns.DnsContextFactory() WARNING: Please consider reporting this to the maintainers of org.infinispan.commons.util.Util WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 15:38:15,613 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [data-grid-source-0-25830|2] (3) [data-grid-source-0-25830, data-grid-source-1-23351, backup-6869] 15:38:15,695 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel data-grid-source local address is backup-6869, physical addresses are [10.128.2.151:7800] 15:38:15,896 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=12.1.2.CD20210504-temporary-redhat-00001 timestamp=2021-05-06T15:38:15.894164Z 15:38:16,495 WARN (jgroups-10,backup-6869) [org.infinispan.CLUSTER] ISPN000071: Caught exception when handling command org.infinispan.manager.impl.ReplicableManagerFunctionCommand@e23e7d9 org.infinispan.commons.CacheConfigurationException: ISPN000436: Cache 'someCache' has been requested, but no matching cache configuration exists at org.infinispan.configuration.ConfigurationManager.getConfiguration(ConfigurationManager.java:61) at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:654) at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:644) at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:533) at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:511) at org.infinispan.manager.impl.AbstractDelegatingEmbeddedCacheManager.getCache(AbstractDelegatingEmbeddedCacheManager.java:193) at org.infinispan.manager.impl.UnwrappingEmbeddedCacheManager.getCache(UnwrappingEmbeddedCacheManager.java:25) at org.infinispan.stats.impl.ClusterCacheStatsImpl$DistributedCacheStatsCallable.apply(ClusterCacheStatsImpl.java:481) at org.infinispan.stats.impl.ClusterCacheStatsImpl$DistributedCacheStatsCallable.apply(ClusterCacheStatsImpl.java:471) at org.infinispan.manager.impl.ReplicableManagerFunctionCommand.invokeAsync(ReplicableManagerFunctionCommand.java:39) at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler$ReplicableCommandRunner.run(GlobalInboundInvocationHandler.java:255) at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleReplicableCommand(GlobalInboundInvocationHandler.java:174) at org.infinispan.remoting.inboundhandler.GlobalInboundInvocationHandler.handleFromCluster(GlobalInboundInvocationHandler.java:113) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processRequest(JGroupsTransport.java:1391) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.processMessage(JGroupsTransport.java:1314) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.access$300(JGroupsTransport.java:133) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1458) at org.jgroups.JChannel.up(JChannel.java:784) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:913) at org.jgroups.protocols.FRAG3.up(FRAG3.java:165) at org.jgroups.protocols.FlowControl.up(FlowControl.java:355) at org.jgroups.protocols.FlowControl.up(FlowControl.java:363) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:876) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 15:38:16,601 INFO (jgroups-10,backup-6869) [org.jboss.threads] JBoss Threads version 2.3.3.Final-redhat-00001 15:38:17,107 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager 15:38:18,697 WARN (blocking-thread--p3-t1) [org.infinispan.encoding.impl.StorageConfigurationManager] ISPN000599: Configuration for cache 'someCache' does not define the encoding for keys or values. If you use operations that require data conversion or queries, you should configure the cache with a specific MediaType for keys or values. 15:38:19,293 INFO (main) [org.infinispan.server.core.RequestTracer] OpenTracing integration is disabled 15:38:19,699 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Started connector HotRod (internal) 15:38:20,109 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 15:38:20,706 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SINGLE_PORT (default) listening on 10.128.2.151:11222 15:38:20,706 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'backup-6869' listening on https://10.128.2.151:11222 15:38:20,799 INFO (main) [org.infinispan.SERVER] ISPN080018: Started connector REST (internal) 15:38:20,801 INFO (main) [org.infinispan.SERVER] ISPN080004: Connector SINGLE_PORT (admin) listening on 10.128.2.151:11223 15:38:20,801 INFO (main) [org.infinispan.SERVER] ISPN080034: Server 'backup-6869' listening on http://10.128.2.151:11223 15:38:21,007 INFO (main) [org.infinispan.SERVER] ISPN080001: Red Hat Data Grid Server 8.2.0.GA started in 15911ms 15:38:26,600 INFO (jgroups-14,backup-6869) [org.infinispan.server.core.backup.BackupManagerImpl] ISPN005043: Starting backup 'backup' 15:38:27,303 INFO (non-blocking-thread--p2-t2) [org.infinispan.server.core.backup.BackupManagerImpl] ISPN005044: Backup file created 'backup.zip' 15:38:29,689 INFO (jgroups-15,backup-6869) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [data-grid-source-1-23351|3] (2) [data-grid-source-1-23351, backup-6869] 15:38:29,694 INFO (jgroups-15,backup-6869) [org.infinispan.CLUSTER] ISPN100001: Node data-grid-source-0-25830 left the cluster 15:38:29,900 ERROR (non-blocking-thread--p2-t1) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand on Cache 'org.infinispan.LOCKS', writing keys [] org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=BackupManagerImpl-backup}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:260) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadReturn(PartitionHandlingInterceptor.java:162) at org.infinispan.interceptors.ExceptionSyncInvocationStage.andFinally(ExceptionSyncInvocationStage.java:49) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:162) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadCommand(PartitionHandlingInterceptor.java:140) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitGetKeyValueCommand(PartitionHandlingInterceptor.java:131) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:27) at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.infinispan.util.concurrent.ConditionFuture.checkConditions(ConditionFuture.java:139) at org.infinispan.util.concurrent.ConditionFuture.update(ConditionFuture.java:93) at org.infinispan.statetransfer.StateTransferLockImpl.notifyTransactionDataReceived(StateTransferLockImpl.java:104) at org.infinispan.statetransfer.StateConsumerImpl.lambda$onTopologyUpdate$5(StateConsumerImpl.java:444) at org.infinispan.util.concurrent.CompletionStages.handleAndCompose(CompletionStages.java:135) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:440) at org.infinispan.statetransfer.StateTransferManagerImpl.updateProviderAndConsumer(StateTransferManagerImpl.java:208) at org.infinispan.statetransfer.StateTransferManagerImpl.lambda$doTopologyUpdate$0(StateTransferManagerImpl.java:195) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:194) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:70) at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:125) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$doHandleTopologyUpdate$12(LocalTopologyManagerImpl.java:397) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:386) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$8(LocalTopologyManagerImpl.java:310) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$orderOnCache$24(LocalTopologyManagerImpl.java:735) at org.infinispan.util.concurrent.ActionSequencer.safeNonBlockingCall(ActionSequencer.java:57) at org.infinispan.util.concurrent.ActionSequencer.access$400(ActionSequencer.java:32) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.run(ActionSequencer.java:226) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.apply(ActionSequencer.java:219) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.apply(ActionSequencer.java:179) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 15:38:29,903 ERROR (jgroups-15,backup-6869) [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] ISPN000405: Caught exception while invoking a cache manager listener! org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [org.infinispan.partitionhandling.AvailabilityException] while invoking method [public void org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener.viewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener@421cbd46 at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:430) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListener(CacheManagerNotifierImpl.java:157) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListeners(CacheManagerNotifierImpl.java:84) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:103) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.receiveClusterView(JGroupsTransport.java:734) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445) at org.jgroups.JChannel.up(JChannel.java:765) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911) at org.jgroups.protocols.FRAG3.up(FRAG3.java:151) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:715) at org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65) at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:113) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:930) at org.jgroups.stack.Protocol.up(Protocol.java:333) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:308) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:450) at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:947) at org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:881) at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:855) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:682) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.protocols.TP.passBatchUp(TP.java:1426) at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:147) at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=BackupManagerImpl-backup}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:260) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadReturn(PartitionHandlingInterceptor.java:162) at org.infinispan.interceptors.ExceptionSyncInvocationStage.andFinally(ExceptionSyncInvocationStage.java:49) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:162) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadCommand(PartitionHandlingInterceptor.java:140) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitGetKeyValueCommand(PartitionHandlingInterceptor.java:131) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.BaseAsyncInterceptor.lambda$new$0(BaseAsyncInterceptor.java:27) at org.infinispan.interceptors.InvocationSuccessFunction.apply(InvocationSuccessFunction.java:25) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.invokeQueuedHandlers(QueueAsyncInvocationStage.java:124) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:87) at org.infinispan.interceptors.impl.QueueAsyncInvocationStage.accept(QueueAsyncInvocationStage.java:33) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at org.infinispan.util.concurrent.ConditionFuture.checkConditions(ConditionFuture.java:139) at org.infinispan.util.concurrent.ConditionFuture.update(ConditionFuture.java:93) at org.infinispan.statetransfer.StateTransferLockImpl.notifyTransactionDataReceived(StateTransferLockImpl.java:104) at org.infinispan.statetransfer.StateConsumerImpl.lambda$onTopologyUpdate$5(StateConsumerImpl.java:444) at org.infinispan.util.concurrent.CompletionStages.handleAndCompose(CompletionStages.java:135) at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:440) at org.infinispan.statetransfer.StateTransferManagerImpl.updateProviderAndConsumer(StateTransferManagerImpl.java:208) at org.infinispan.statetransfer.StateTransferManagerImpl.lambda$doTopologyUpdate$0(StateTransferManagerImpl.java:195) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:194) at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:70) at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:125) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$doHandleTopologyUpdate$12(LocalTopologyManagerImpl.java:397) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143) at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:386) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$8(LocalTopologyManagerImpl.java:310) at org.infinispan.topology.LocalTopologyManagerImpl.lambda$orderOnCache$24(LocalTopologyManagerImpl.java:735) at org.infinispan.util.concurrent.ActionSequencer.safeNonBlockingCall(ActionSequencer.java:57) at org.infinispan.util.concurrent.ActionSequencer.access$400(ActionSequencer.java:32) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.run(ActionSequencer.java:226) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.apply(ActionSequencer.java:219) at org.infinispan.util.concurrent.ActionSequencer$SequenceEntry.apply(ActionSequencer.java:179) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) ... 3 more Suppressed: org.infinispan.commons.util.logging.TraceException at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:39) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246) at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297) at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101) at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:543) at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:526) at org.infinispan.cache.impl.DecoratedCache.containsKey(DecoratedCache.java:511) at org.infinispan.cache.impl.AbstractDelegatingCache.containsKey(AbstractDelegatingCache.java:384) at org.infinispan.cache.impl.EncoderCache.containsKey(EncoderCache.java:601) at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.isDefined(EmbeddedClusteredLockManager.java:157) at org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener.viewChange(ClusteredLockImpl.java:335) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$0(AbstractListenerImpl.java:410) at org.infinispan.security.Security.doAs(Security.java:158) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:407) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListener(CacheManagerNotifierImpl.java:157) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListeners(CacheManagerNotifierImpl.java:84) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:103) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.receiveClusterView(JGroupsTransport.java:734) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445) at org.jgroups.JChannel.up(JChannel.java:765) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911) at org.jgroups.protocols.FRAG3.up(FRAG3.java:151) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:715) at org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65) at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:113) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:930) at org.jgroups.stack.Protocol.up(Protocol.java:333) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:308) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:450) at org.jgroups.protocols.pbcast.NAKACK2.deliverBatch(NAKACK2.java:947) at org.jgroups.protocols.pbcast.NAKACK2.removeAndDeliver(NAKACK2.java:881) at org.jgroups.protocols.pbcast.NAKACK2.handleMessages(NAKACK2.java:855) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:682) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:197) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.stack.Protocol.up(Protocol.java:341) at org.jgroups.protocols.TP.passBatchUp(TP.java:1426) at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.passBatchUp(MaxOneThreadPerSender.java:284) at org.jgroups.util.SubmitToThreadPool$BatchHandler.run(SubmitToThreadPool.java:147) at org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:273) ... 3 more 15:38:29,997 INFO (jgroups-15,backup-6869) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel data-grid-source: [backup-6869|4] (1) [backup-6869] 15:38:29,998 INFO (jgroups-15,backup-6869) [org.infinispan.CLUSTER] ISPN100001: Node data-grid-source-1-23351 left the cluster 15:38:30,000 ERROR (jgroups-15,backup-6869) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command GetKeyValueCommand on Cache 'org.infinispan.LOCKS', writing keys [] org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=BackupManagerImpl-backup}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:260) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadReturn(PartitionHandlingInterceptor.java:162) at org.infinispan.interceptors.ExceptionSyncInvocationStage.andFinally(ExceptionSyncInvocationStage.java:49) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:162) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadCommand(PartitionHandlingInterceptor.java:140) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitGetKeyValueCommand(PartitionHandlingInterceptor.java:131) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:188) at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:197) at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:172) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:147) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:123) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:112) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:89) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:112) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:51) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:244) at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297) at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101) at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:543) at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:526) at org.infinispan.cache.impl.DecoratedCache.containsKey(DecoratedCache.java:511) at org.infinispan.cache.impl.AbstractDelegatingCache.containsKey(AbstractDelegatingCache.java:384) at org.infinispan.cache.impl.EncoderCache.containsKey(EncoderCache.java:601) at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.isDefined(EmbeddedClusteredLockManager.java:157) at org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener.viewChange(ClusteredLockImpl.java:335) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$0(AbstractListenerImpl.java:410) at org.infinispan.security.Security.doAs(Security.java:158) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:407) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListener(CacheManagerNotifierImpl.java:157) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListeners(CacheManagerNotifierImpl.java:84) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:103) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.receiveClusterView(JGroupsTransport.java:734) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445) at org.jgroups.JChannel.up(JChannel.java:765) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911) at org.jgroups.protocols.FRAG3.up(FRAG3.java:151) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:715) at org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65) at org.jgroups.protocols.pbcast.GMS.castViewChangeAndSendJoinRsps(GMS.java:592) at org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:198) at org.jgroups.protocols.pbcast.GMS.process(GMS.java:1296) at org.jgroups.protocols.pbcast.ViewHandler.removeAndProcess(ViewHandler.java:280) at org.jgroups.protocols.pbcast.ViewHandler.process(ViewHandler.java:244) at org.jgroups.protocols.pbcast.ViewHandler.add(ViewHandler.java:64) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:893) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 15:38:30,003 WARN (TcpServer.Acceptor[7800]-2,backup-6869) [org.jgroups.protocols.TCP] JGRP000006: failed accepting connection from peer Socket[addr=/10.129.3.108,port=51591,localport=7800]: java.io.EOFException java.io.EOFException at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202) at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:247) at org.jgroups.blocks.cs.TcpConnection.(TcpConnection.java:53) at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:129) at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:113) at java.base/java.lang.Thread.run(Thread.java:829) 15:38:30,002 ERROR (jgroups-15,backup-6869) [org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl] ISPN000405: Caught exception while invoking a cache manager listener! org.infinispan.commons.CacheListenerException: ISPN000280: Caught exception [org.infinispan.partitionhandling.AvailabilityException] while invoking method [public void org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener.viewChange(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent)] on listener instance: org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener@421cbd46 at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:430) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:450) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListener(CacheManagerNotifierImpl.java:157) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.invokeListeners(CacheManagerNotifierImpl.java:84) at org.infinispan.notifications.cachemanagerlistener.CacheManagerNotifierImpl.notifyViewChange(CacheManagerNotifierImpl.java:103) at org.infinispan.remoting.transport.jgroups.JGroupsTransport.receiveClusterView(JGroupsTransport.java:734) at org.infinispan.remoting.transport.jgroups.JGroupsTransport$ChannelCallbacks.up(JGroupsTransport.java:1445) at org.jgroups.JChannel.up(JChannel.java:765) at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:911) at org.jgroups.protocols.FRAG3.up(FRAG3.java:151) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.FlowControl.up(FlowControl.java:342) at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:715) at org.jgroups.protocols.pbcast.ServerGmsImpl.handleViewChange(ServerGmsImpl.java:65) at org.jgroups.protocols.pbcast.GMS.castViewChangeAndSendJoinRsps(GMS.java:592) at org.jgroups.protocols.pbcast.CoordGmsImpl.handleMembershipChange(CoordGmsImpl.java:198) at org.jgroups.protocols.pbcast.GMS.process(GMS.java:1296) at org.jgroups.protocols.pbcast.ViewHandler.removeAndProcess(ViewHandler.java:280) at org.jgroups.protocols.pbcast.ViewHandler.process(ViewHandler.java:244) at org.jgroups.protocols.pbcast.ViewHandler.add(ViewHandler.java:64) at org.jgroups.protocols.pbcast.GMS.up(GMS.java:893) at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:254) at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1048) at org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:771) at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:752) at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:405) at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:592) at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:132) at org.jgroups.protocols.FailureDetection.up(FailureDetection.java:186) at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:254) at org.jgroups.protocols.MERGE3.up(MERGE3.java:281) at org.jgroups.protocols.Discovery.up(Discovery.java:300) at org.jgroups.protocols.TP.passMessageUp(TP.java:1400) at org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:98) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.infinispan.partitionhandling.AvailabilityException: ISPN000306: Key 'ClusteredLockKey{name=BackupManagerImpl-backup}' is not available. Not all owners are in this partition at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.doCheck(PartitionHandlingManagerImpl.java:260) at org.infinispan.partitionhandling.impl.PartitionHandlingManagerImpl.checkRead(PartitionHandlingManagerImpl.java:103) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadReturn(PartitionHandlingInterceptor.java:162) at org.infinispan.interceptors.ExceptionSyncInvocationStage.andFinally(ExceptionSyncInvocationStage.java:49) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndFinally(BaseAsyncInterceptor.java:162) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.handleDataReadCommand(PartitionHandlingInterceptor.java:140) at org.infinispan.partitionhandling.impl.PartitionHandlingInterceptor.visitGetKeyValueCommand(PartitionHandlingInterceptor.java:131) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndHandle(BaseAsyncInterceptor.java:188) at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.handleReadCommand(BaseStateTransferInterceptor.java:197) at org.infinispan.interceptors.impl.BaseStateTransferInterceptor.visitGetKeyValueCommand(BaseStateTransferInterceptor.java:172) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:147) at org.infinispan.interceptors.impl.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:123) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:59) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:112) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:128) at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:89) at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:61) at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:55) at org.infinispan.interceptors.DDAsyncInterceptor.visitGetKeyValueCommand(DDAsyncInterceptor.java:112) at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:34) at org.infinispan.interceptors.DDAsyncInterceptor.visitCommand(DDAsyncInterceptor.java:51) at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:244) at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:297) at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:101) at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:543) at org.infinispan.cache.impl.CacheImpl.containsKey(CacheImpl.java:526) at org.infinispan.cache.impl.DecoratedCache.containsKey(DecoratedCache.java:511) at org.infinispan.cache.impl.AbstractDelegatingCache.containsKey(AbstractDelegatingCache.java:384) at org.infinispan.cache.impl.EncoderCache.containsKey(EncoderCache.java:601) at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.isDefined(EmbeddedClusteredLockManager.java:157) at org.infinispan.lock.impl.lock.ClusteredLockImpl$ClusterChangeListener.viewChange(ClusteredLockImpl.java:335) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$0(AbstractListenerImpl.java:410) at org.infinispan.security.Security.doAs(Security.java:158) at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:407) ... 36 more 15:38:30,097 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=someCache]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,099 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,100 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.CONFIG]ISPN100007: After merge (or coordinator change), recovered members [backup-6869] with topology id 15 15:38:30,195 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,196 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,197 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,198 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 12 15:38:30,204 ERROR (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN000319: After merge (or coordinator change), cache still hasn't recovered all its data and must stay in degraded mode. Current members are [backup-6869], lost members are [data-grid-source-1-23351], stable members are [data-grid-source-1-23351, backup-6869] 15:38:30,205 INFO (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.LOCKS]ISPN100007: After merge (or coordinator change), recovered members [data-grid-source-1-23351, backup-6869] with topology id 14 15:38:30,217 FATAL (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___hotRodTopologyCache_hotrod-default]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,217 FATAL (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___protobuf_metadata]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,218 FATAL (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=org.infinispan.COUNTER]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,219 FATAL (non-blocking-thread--p2-t2) [org.infinispan.CLUSTER] [Context=someCache]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,219 FATAL (non-blocking-thread--p2-t1) [org.infinispan.CLUSTER] [Context=___script_cache]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,219 FATAL (non-blocking-thread--p2-t2) [org.infinispan.CLUSTER] [Context=org.infinispan.CLIENT_SERVER_TX_TABLE]ISPN000313: Lost data because of abrupt leavers [data-grid-source-1-23351] 15:38:30,955 INFO (pool-7-thread-1) [org.infinispan.SERVER] ISPN080002: Red Hat Data Grid Server stopping 15:38:31,066 INFO (pool-7-thread-1) [org.infinispan.manager.DefaultCacheManager] Stopping cache manager on backup-6869 15:38:31,069 ERROR (blocking-thread--p3-t14) [org.infinispan.stats.impl.ClusterCacheStatsImpl] Could not execute cluster wide cache stats operation java.lang.NullPointerException at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:497) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:492) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:369) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:141) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:117) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:210) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsLong(AbstractClusterStats.java:201) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getOffHeapMemoryUsed(ClusterCacheStatsImpl.java:342) at org.infinispan.factories.impl.MBeanMetadata$AttributeMetadata.lambda$getter$0(MBeanMetadata.java:137) at org.infinispan.metrics.impl.MetricsCollector.lambda$registerMetrics$0(MetricsCollector.java:139) at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492) at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192) at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158) at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73) at org.infinispan.rest.resources.MetricsResource.lambda$metrics$1(MetricsResource.java:80) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:829) 15:38:31,070 WARN (blocking-thread--p3-t14) [io.smallrye.metrics] Unable to export metric cache_manager_default_cache_someCache_cluster_cache_stats_off_heap_memory_used org.infinispan.commons.CacheException: Could not execute cluster wide cache stats operation at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:120) at org.infinispan.stats.impl.AbstractClusterStats.getStat(AbstractClusterStats.java:210) at org.infinispan.stats.impl.AbstractClusterStats.getStatAsLong(AbstractClusterStats.java:201) at org.infinispan.stats.impl.ClusterCacheStatsImpl.getOffHeapMemoryUsed(ClusterCacheStatsImpl.java:342) at org.infinispan.factories.impl.MBeanMetadata$AttributeMetadata.lambda$getter$0(MBeanMetadata.java:137) at org.infinispan.metrics.impl.MetricsCollector.lambda$registerMetrics$0(MetricsCollector.java:139) at io.smallrye.metrics.exporters.OpenMetricsExporter.createSimpleValueLine(OpenMetricsExporter.java:492) at io.smallrye.metrics.exporters.OpenMetricsExporter.exposeEntries(OpenMetricsExporter.java:192) at io.smallrye.metrics.exporters.OpenMetricsExporter.getEntriesForScope(OpenMetricsExporter.java:158) at io.smallrye.metrics.exporters.OpenMetricsExporter.exportAllScopes(OpenMetricsExporter.java:109) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:116) at io.smallrye.metrics.MetricsRequestHandler.handleRequest(MetricsRequestHandler.java:73) at org.infinispan.rest.resources.MetricsResource.lambda$metrics$1(MetricsResource.java:80) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.NullPointerException at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:497) at org.infinispan.cache.impl.CacheImpl.size(CacheImpl.java:492) at org.infinispan.cache.impl.AbstractDelegatingCache.size(AbstractDelegatingCache.java:369) at org.infinispan.stats.impl.ClusterCacheStatsImpl.updateStats(ClusterCacheStatsImpl.java:141) at org.infinispan.stats.impl.AbstractClusterStats.fetchClusterWideStatsIfNeeded(AbstractClusterStats.java:117) ... 18 more 15:38:31,104 INFO (pool-7-thread-1) [org.infinispan.CLUSTER] ISPN000080: Disconnecting JGroups channel data-grid-source 15:38:31,110 INFO (pool-7-thread-1) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=12.1.2.CD20210504-temporary-redhat-00001 timestamp=2021-05-06T15:38:31.109967Z 15:38:31,113 INFO (pool-7-thread-1) [org.infinispan.SERVER] ISPN080003: Red Hat Data Grid Server stopped 15:38:32,063 INFO (Thread-1) [org.infinispan.SERVER] ISPN080002: Red Hat Data Grid Server stopping 15:38:32,063 INFO (Thread-1) [org.infinispan.SERVER] ISPN080003: Red Hat Data Grid Server stopped