Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

Oracle 2.5.0.Beta1

What is the connector configuration?

"offset.storage.jdbc.url", connectionSetupFromDbd.getUrl());"offset.storage.jdbc.user", connectionSetupFromDbd.getUid());"offset.storage.jdbc.password", encryptedPwd);"offset.storage.jdbc.offset.table.name", DebeziumStorageManager.TABLE_DEBEZIUM_OFFSET_STORAGE);
"schema.history.internal", "io.debezium.storage.jdbc.history.JdbcSchemaHistory");"schema.history.internal.jdbc.url", connectionSetupFromDbd.getUrl());"schema.history.internal.jdbc.user", connectionSetupFromDbd.getUid());"schema.history.internal.jdbc.password", encryptedPwd);"schema.history.internal.jdbc.schema.history.table.name", DebeziumStorageManager.TABLE_DEBEZIUM_DATABA"schema.history.internal.jdbc.schema.history.table.ddl", "CREATE TABLE %s" +
"(" +
"id VARCHAR(36) NOT NULL," +
"history_data CLOB," +
"history_data_seq INTEGER," +
"record_insert_ts TIMESTAMP NOT NULL," +
"record_insert_seq INTEGER NOT NULL" +
")");"schema.history.internal.jdbc.schema.history.table.select", "SELECT * FROM %s where ROWNUM <= 1");
 

What is the captured database version and mode of depoyment?

(E.g. on-premises, with a specific cloud provider, etc.)

embedded debezium

What behaviour do you expect?

Consistency that if debezium mining works, the offset is being saved into the DB

What behaviour do you see?

There was an intermittent database problem and debezium oracle logminer connector was able to recover. However the JdbcOffsetBackingStore did not recover and was not updating the offset.

There was an error thrown which did not interrupt the embedded engine from running. So we have some erros in the logs but debezium keeps sending files for days and upon restart, the offset is 2 days old.

We use record commiter manually:
private DebeziumEngine.RecordCommitter<ChangeEvent<SourceRecord, SourceRecord>> changeEventRecordCommitter;
The problem I think that Debezium has some mechanism for DB connection recovery but that is not propagated to the Offset store. Over there, the connection is retrieved during startup and never updated.

Flush of EmbeddedEngine{id=debezium-oracle:debezium_interfaceId_99328} offsets threw an unexpected exception: java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: java.sql.SQLRecoverableException: Closed Connection at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) at io.debezium.embedded.EmbeddedEngine.commitOffsets(EmbeddedEngine.java:912) at io.debezium.embedded.EmbeddedEngine.maybeFlush(EmbeddedEngine.java:888) at io.debezium.embedded.EmbeddedEngine$4.markBatchFinished(EmbeddedEngine.java:827) at io.debezium.embedded.ConvertingEngineBuilder$1.markBatchFinished(ConvertingEngineBuilder.java:113) at com.onsemi.middleware.cdc.embed.DebeziumEmbedServer.onApplicationEvent(DebeziumEmbedServer.java:241) at com.onsemi.middleware.cdc.embed.DebeziumEmbedServer.onApplicationEvent(DebeziumEmbedServer.java:45) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378) at com.onsemi.middleware.cdc.target.EntityFile.sendFile(EntityFile.java:150) at com.onsemi.middleware.cdc.target.FTPLikeTarget.sendAllThatHaveData(FTPLikeTarget.java:110) at com.onsemi.middleware.cdc.target.FTPLikeTarget.processDml(FTPLikeTarget.java:68) at com.onsemi.middleware.cdc.target.FTPLikeTarget.process(FTPLikeTarget.java:53) at org.apache.camel.support.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:65) at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:471) at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:193) at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64) at org.apache.camel.processor.Pipeline.process(Pipeline.java:184) at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398) at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:210) at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:76) at java.base/java.util.TimerThread.mainLoop(Timer.java:556) at java.base/java.util.TimerThread.run(Timer.java:506) Caused by: org.apache.kafka.connect.errors.ConnectException: java.sql.SQLRecoverableException: Closed Connection at io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore.save(JdbcOffsetBackingStore.java:138) at io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore$1.call(JdbcOffsetBackingStore.java:203) at io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore$1.call(JdbcOffsetBackingStore.java:194) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.sql.SQLRecoverableException: Closed Connection at oracle.jdbc.driver.PhysicalConnection.prepareStatementInternal(PhysicalConnection.java:1746) at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:1721) at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:1648) at io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore.save(JdbcOffsetBackingStore.java:112) ... 6 more

Do you see the same behaviour using the latest relesead Debezium version?

(Ideally, also verify with latest Alpha/Beta/CR version)

yes

Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide DEBUG/TRACE level log)

yes

How to reproduce the issue using our tutorial deployment?

bring mined DB down, bring it up, wait for log mining to recover, observe the offset store throws an exceptions