Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 31.0.0.Final
Affects Version/s: 30.0.0.Final
Component/s: Clustering
Labels:
None

Affects Testing:

Regression
Git Pull Request:
https://github.com/wildfly/wildfly/pull/17446
CDW blocker:
[QE] How to address?:
---
[QE] Why QE missed?:
---

Scenario: fail-over scenario where we have the usual 4 nodes cluster where each node is configured to store session data into an invalidation-cache backed by an Oracle 19c RAC Database;

This is a quite common scenario where session data is offloaded to some relational database, in facts we test it with all supported databases;

Each node is running on JDK17 (but we spotted java.io.InvalidClassException with JDK11 too);
Each node is configured as in the following:

embed-server --server-config=standalone-ha.xml
/subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload:add()
data-source add --name=testDS --jndi-name=java:jboss/datasources/testDS --driver-name=oracle-connector.jar --connection-url=jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-19c-rac-01)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=oracle-19c-rac-02)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=...))) --enabled=true --jta=true --use-java-context=true --transaction-isolation=TRANSACTION_READ_COMMITTED --min-pool-size=1 --max-pool-size=5 --pool-prefill=true --user-name=... --password=... --prepared-statements-cache-size=32 --share-prepared-statements=true
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc:add(data-source=testDS,fetch-state=false,passivation=false,purge=false,shared=true,dialect=ORACLE){allow-resource-service-restart=true}
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.name,value=id)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.name,value=datum)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.name,value=version)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=id-column.type,value=VARCHAR2(255))
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=timestamp-column.type,value=NUMBER)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=data-column.type,value=BLOB)
/subsystem=infinispan/cache-container=web/invalidation-cache=offload/store=jdbc/table=string:write-attribute(name=prefix, value=i){allow-resource-service-restart=true}
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=offload)
/subsystem=transactions:write-attribute(name=node-identifier,value=wildfly1)

The 4 nodes are started and the cluster is formed; then, each node is failed in sequence by shutting down and restarting WildFly;

When node1 is stopped:

node1 is started
node2 is started
node3 is started
node4 is started
node1 is shut down

We observed the following ERROR on node4 (and, some times later, on node3 too) (complete logs in [^server-logs.zip] ):

2023-11-22 14:35:39,577 ERROR [org.infinispan.PERSISTENCE] (SchedulerTopologyChangeListener - 3) ISPN008009: I/O error while unmarshalling from stream: java.io.IOException: ORA-01555: snapshot too old: rollback segment number  with name "" too small
ORA-22924: snapshot too old

	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBlobInputStream.needBytes(OracleBlobInputStream.java:239)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBufferedStream.needBytes(OracleBufferedStream.java:127)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBufferedStream.readInternal(OracleBufferedStream.java:187)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBufferedStream.read(OracleBufferedStream.java:171)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.impl.TagReaderImpl$InputStreamDecoder.isMarkDone(TagReaderImpl.java:1022)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.impl.TagReaderImpl$InputStreamDecoder.isAtEnd(TagReaderImpl.java:1008)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.impl.TagReaderImpl$Decoder.readTag(TagReaderImpl.java:305)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.impl.TagReaderImpl.readTag(TagReaderImpl.java:97)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.WrappedMessage.readMessage(WrappedMessage.java:365)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.WrappedMessage.read(WrappedMessage.java:351)
	at org.infinispan.protostream@4.6.5.Final-redhat-00003//org.infinispan.protostream.ProtobufUtil.fromWrappedStream(ProtobufUtil.java:137)
	at org.infinispan@14.0.17.Final-redhat-00002//org.infinispan.marshall.protostream.impl.AbstractInternalProtoStreamMarshaller.readObject(AbstractInternalProtoStreamMarshaller.java:120)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.JdbcUtil.unmarshall(JdbcUtil.java:68)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.impl.table.AbstractTableManager.entryFromResultSet(AbstractTableManager.java:630)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.sql.BaseTableOperations$ResultSetEntryIterator.getNext(BaseTableOperations.java:301)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.sql.BaseTableOperations$ResultSetEntryIterator.getNext(BaseTableOperations.java:286)
	at org.infinispan.commons@14.0.17.Final-redhat-00002//org.infinispan.commons.util.AbstractIterator.hasNext(AbstractIterator.java:26)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$IteratorSubscription.slowPath(FlowableFromIterable.java:253)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable$BaseRangeSubscription.request(FlowableFromIterable.java:131)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.request(FlowableDoFinally.java:108)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.request(FlowableUsing.java:153)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.request(BasicFuseableSubscriber.java:153)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.request(FlowableUsing.java:153)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.subscriptions.SubscriptionArbiter.setSubscription(SubscriptionArbiter.java:87)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatArray$ConcatArraySubscriber.onSubscribe(FlowableConcatArray.java:72)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onSubscribe(FlowableUsing.java:98)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.subscribers.BasicFuseableSubscriber.onSubscribe(BasicFuseableSubscriber.java:67)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onSubscribe(FlowableUsing.java:98)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableDoFinally$DoFinallySubscriber.onSubscribe(FlowableDoFinally.java:79)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribe(FlowableFromIterable.java:69)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableFromIterable.subscribeActual(FlowableFromIterable.java:47)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableDoFinally.subscribeActual(FlowableDoFinally.java:47)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15959)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing.subscribeActual(FlowableUsing.java:73)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:38)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15959)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing.subscribeActual(FlowableUsing.java:73)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:15959)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatArray$ConcatArraySubscriber.onComplete(FlowableConcatArray.java:141)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableConcatArray.subscribeActual(FlowableConcatArray.java:41)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableMap.subscribeActual(FlowableMap.java:36)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableFilter.subscribeActual(FlowableFilter.java:38)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.FlowableOnErrorNext.subscribeActual(FlowableOnErrorNext.java:39)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.core.Flowable.subscribe(Flowable.java:16013)
	at io.reactivex.rxjava3.rxjava@3.1.6.redhat-00001//io.reactivex.rxjava3.internal.operators.flowable.BlockingFlowableIterable.iterator(BlockingFlowableIterable.java:42)
	at org.infinispan.commons@14.0.17.Final-redhat-00002//org.infinispan.commons.util.Closeables.iterator(Closeables.java:238)
	at org.infinispan@14.0.17.Final-redhat-00002//org.infinispan.stream.impl.DistributedCacheStream.iterator(DistributedCacheStream.java:370)
	at org.infinispan@14.0.17.Final-redhat-00002//org.infinispan.util.AbstractDelegatingCacheStream.iterator(AbstractDelegatingCacheStream.java:182)
	at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-00010//org.wildfly.clustering.ee.infinispan.scheduler.ScheduleLocalKeysTask.accept(ScheduleLocalKeysTask.java:61)
	at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-00010//org.wildfly.clustering.ee.infinispan.scheduler.ScheduleLocalKeysTask.accept(ScheduleLocalKeysTask.java:42)
	at org.wildfly.clustering.ee.infinispan@8.0.0.GA-redhat-00010//org.wildfly.clustering.ee.infinispan.scheduler.SchedulerTopologyChangeListener.lambda$topologyChanged$2(SchedulerTopologyChangeListener.java:121)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
	at java.base/java.lang.Thread.run(Thread.java:833)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.sql.SQLException: ORA-01555: snapshot too old: rollback segment number  with name "" too small
ORA-22924: snapshot too old

	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:456)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:451)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4C8TTILob.processError(T4C8TTILob.java:838)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4C8TTILob.read(T4C8TTILob.java:159)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.T4CConnection.getBytes(T4CConnection.java:3647)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBlob.getBytes(OracleBlob.java:382)
	at deployment.oracle-connector.jar//oracle.jdbc.driver.OracleBlobInputStream.needBytes(OracleBlobInputStream.java:220)
	... 66 more

When node3 is stopped:

node1 is started
node2 is started
node3 is started
node4 is started
node1 is shut down
node1 is started
node2 is shut down
node2 is started
node3 is shut down

We observed the following WARN on node1 (and, some times later, on the other nodes too):

2023-11-22 14:44:05,116 WARN  [org.wildfly.clustering.server.infinispan] (thread-34,ejb,wildfly1) java.sql.SQLException from [Module "java.base" version 17.0.7 from ModuleLoader@34d7485d for finders [JDK Module Finder]]: java.io.InvalidClassException: java.sql.SQLException from [Module "java.base" version 17.0.7 from ModuleLoader@34d7485d for finders [JDK Module Finder]]
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.LoadedClassField.readFrom(LoadedClassField.java:68)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.LoadedClassField.readFrom(LoadedClassField.java:35)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ClassMarshaller.readFrom(ClassMarshaller.java:52)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ClassMarshaller.readFrom(ClassMarshaller.java:35)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readObject(DefaultProtoStreamReader.java:92)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ExceptionMarshaller.readFrom(ExceptionMarshaller.java:68)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ExceptionMarshaller.readFrom(ExceptionMarshaller.java:38)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readObject(DefaultProtoStreamReader.java:92)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.TypedObjectMarshaller.readFrom(TypedObjectMarshaller.java:49)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:51)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:36)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readObject(DefaultProtoStreamReader.java:92)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readAny(DefaultProtoStreamReader.java:66)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ProtoStreamReader.readAny(ProtoStreamReader.java:49)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ExceptionMarshaller.readFrom(ExceptionMarshaller.java:74)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ExceptionMarshaller.readFrom(ExceptionMarshaller.java:38)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.DefaultProtoStreamReader.readObject(DefaultProtoStreamReader.java:92)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.TypedObjectMarshaller.readFrom(TypedObjectMarshaller.java:49)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:51)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.AnyMarshaller.readFrom(AnyMarshaller.java:36)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.ProtoStreamByteBufferMarshaller.readFrom(ProtoStreamByteBufferMarshaller.java:85)
	at org.wildfly.clustering.marshalling.spi@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.spi.ByteBufferMarshaller.read(ByteBufferMarshaller.java:59)
	at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-00010//org.wildfly.clustering.server.infinispan.dispatcher.CommandDispatcherRequestCorrelator.readPayload(CommandDispatcherRequestCorrelator.java:89)
	at org.wildfly.clustering.server.infinispan@8.0.0.GA-redhat-00010//org.wildfly.clustering.server.infinispan.dispatcher.CommandDispatcherRequestCorrelator.dispatch(CommandDispatcherRequestCorrelator.java:68)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:247)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:498)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.JChannel.up(JChannel.java:733)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:936)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FORK.up(FORK.java:141)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FRAG2.up(FRAG2.java:139)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:253)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FlowControl.up(FlowControl.java:261)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.pbcast.GMS.up(GMS.java:845)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:226)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1083)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.UNICAST3.addMessage(UNICAST3.java:822)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:804)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:453)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:680)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.VERIFY_SUSPECT2.up(VERIFY_SUSPECT2.java:105)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:180)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.FD_SOCK2.up(FD_SOCK2.java:188)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.MERGE3.up(MERGE3.java:274)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.Discovery.up(Discovery.java:294)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.stack.Protocol.up(Protocol.java:340)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.protocols.TP.passMessageUp(TP.java:1184)
	at org.jgroups@5.2.18.Final-redhat-00001//org.jgroups.util.SubmitToThreadPool$SingleMessageHandler.run(SubmitToThreadPool.java:107)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: java.sql.SQLException from [Module "java.base" version 17.0.7 from ModuleLoader@34d7485d for finders [JDK Module Finder]]
	at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:200)
	at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
	at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
	at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
	at org.wildfly.clustering.marshalling.protostream@8.0.0.GA-redhat-00010//org.wildfly.clustering.marshalling.protostream.LoadedClassField.readFrom(LoadedClassField.java:66)
	... 51 more

On node1, we observe also the following WARN:

2023-11-22 14:45:39,684 WARN  [org.infinispan.persistence.manager.PersistenceManagerImpl] (non-blocking-thread--p6-t4) ISPN000677: Store 'org.jboss.as.clustering.infinispan.persistence.jdbc.JDBCStore'#isAvailable completed Exceptionally: java.util.concurrent.CompletionException: org.infinispan.persistence.spi.PersistenceException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.infinispan.commons@14.0.17.Final-redhat-00002//org.infinispan.commons.util.concurrent.CompletableFutures.asCompletionException(CompletableFutures.java:160)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.impl.BaseJdbcStore.lambda$isAvailable$2(BaseJdbcStore.java:114)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.infinispan.persistence.spi.PersistenceException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:58)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.impl.BaseJdbcStore.lambda$isAvailable$2(BaseJdbcStore.java:110)
	... 8 more
Caused by: java.sql.SQLException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:158)
	at org.jboss.as.connector@8.0.0.GA-redhat-00010//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:56)
	... 9 more
Caused by: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:150)
	... 11 more
Caused by: jakarta.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:579)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	... 14 more

Rigth after node3 is started once again, we observe the following WANR and ERROR on node1:

2023-11-22 14:45:39,686 WARN  [org.infinispan.persistence.manager.PersistenceManagerImpl] (non-blocking-thread--p6-t4) ISPN000678: Persistence is unavailable because of store org.jboss.as.clustering.infinispan.persistence.jdbc.JDBCStore
2023-11-22 14:45:40,086 WARN  [org.infinispan.persistence.manager.PersistenceManagerImpl] (non-blocking-thread--p6-t64) ISPN000677: Store 'org.jboss.as.clustering.infinispan.persistence.jdbc.JDBCStore'#isAvailable completed Exceptionally: java.util.concurrent.CompletionException: org.infinispan.persistence.spi.PersistenceException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.infinispan.commons@14.0.17.Final-redhat-00002//org.infinispan.commons.util.concurrent.CompletableFutures.asCompletionException(CompletableFutures.java:160)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.impl.BaseJdbcStore.lambda$isAvailable$2(BaseJdbcStore.java:114)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.infinispan.persistence.spi.PersistenceException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:58)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.impl.BaseJdbcStore.lambda$isAvailable$2(BaseJdbcStore.java:110)
	... 8 more
Caused by: java.sql.SQLException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:158)
	at org.jboss.as.connector@8.0.0.GA-redhat-00010//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:56)
	... 9 more
Caused by: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:150)
	... 11 more
Caused by: jakarta.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:579)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	... 14 more

2023-11-22 14:45:40,087 WARN  [org.infinispan.persistence.manager.PersistenceManagerImpl] (non-blocking-thread--p6-t64) ISPN000678: Persistence is unavailable because of store org.jboss.as.clustering.infinispan.persistence.jdbc.JDBCStore
2023-11-22 14:45:40,983 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (non-blocking-thread--p6-t21) ISPN000136: Error executing command GetAllCommand on Cache 'cbnc.ear.a.war', writing keys []: org.infinispan.persistence.spi.PersistenceException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:58)
	at org.infinispan.persistence.jdbc@14.0.17.Final-redhat-00002//org.infinispan.persistence.jdbc.common.impl.BaseJdbcStore.lambda$load$3(BaseJdbcStore.java:126)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1363)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
	at org.wildfly.clustering.context@8.0.0.GA-redhat-00010//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.sql.SQLException: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:158)
	at org.jboss.as.connector@8.0.0.GA-redhat-00010//org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64)
	at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00010//org.jboss.as.clustering.infinispan.persistence.jdbc.DataSourceConnectionFactory.getConnection(DataSourceConnectionFactory.java:56)
	... 9 more
Caused by: jakarta.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/testDS
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at org.jboss.ironjacamar.jdbcadapters@3.0.2.Final-redhat-00002//org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:150)
	... 11 more
Caused by: jakarta.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:579)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at org.jboss.ironjacamar.impl@3.0.2.Final-redhat-00002//org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	... 14 more

We were using driver ojdbc10.jar;

clones

JBEAP-26112 (8.0.z) [CLUSTERING]: java.io.InvalidClassException with ORACLE Data store

Closed

is caused by

WFLY-18683 Blocked JDBC store threads prevent shutdown

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates