What Debezium connector do you use and what version?

It is 2.5.4, but here are helpful additional library information we use in our Java Springboot application.

implementation 'io.debezium:debezium-api:2.5.4.Final'
implementation'io.debezium:debezium-embedded:2.5.4.Final'
implementation 'org.apache.kafka:connect-runtime:3.6.1'
implementation 'io.debezium:debezium-connector-oracle:2.5.4.Final'
implementation 'com.oracle.database.jdbc:ojdbc8:21.6.0.0'
implementation 'io.debezium:debezium-storage-kafka:2.5.4.Final'

What is the connector configuration?

connector:
decimalHandlingMode: string
name: <<NAME>>
class: io.debezium.connector.oracle.OracleConnector
offsetTopic: <<OFFSETTOPICNAME>>
offsetTopicNumberOfPartitions: 1
offsetTopicReplicationFactor: 1
heartbeatIntervalInMsecs: 100000
heartbeatQuery: SELECT * FROM MYTABLE WHERE ROWNUM <= 1

What is the captured database version and mode of deployment?

Data base in running on a data center and is an Oracle database of version 19c.

DB Server on Dev Database runs on a 2 node RAC. The Archived log files (which Log Miner reads from) are shared to both nodes in RAC as a shared path.

What behavior do you expect?

Debezium listener should listen smoothly and even if there an error, it should try to recover

What behavior do you see?

It doesn't recover after a ORA-00600 error with the following error seen in Oracle DB logs

VERSION INFORMATION: TNS for Linux: Version 19.0.0.0.0 - Production Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production Version 19.23.0.0.0 Time: 05-SEP-2024 17:42:43 Tracing not turned on. Tns error struct: ns main err code: 12535 TNS-12535: TNS:operation timed out ns secondary err code: 12560 nt main err code: 505 TNS-00505: Operation timed out nt secondary err code: 110 nt OS err code: 0 Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.45.30.19)(PORT=36082)) 2024-09-05T17:42:43.637019-05:00 NET (PID:76125): Possible network disconnect with primary database

Do you see the same behaviour using the latest released Debezium version?

Didn't try yet. Is 2.5.4.Final OK ?

Do you have the connector logs, ideally from start till finish?

Connector logs were working fine, and suddenly gave ORA-00600 at around the same time when we observed the above timeout error in DB. Can explain below.

How to reproduce the issue using our tutorial deployment?

Not always.

DESCRIPTION OF THE ISSUE

Java based Debezium connector with Oracle was streaming smoothly. All of a sudden after some 24 hours, it simply stops and following are the connector logs observed with redacted message

metadata=Metadata(originalPublishTimestamp=2024-09-05T22:42:40Z, source=UMD, topic=fdc-prod-datascience-combined), timestamp=2024-09-05T22:42:40.883Z)] with offset=[1023163507] partition=[0] key=[b8a106c6-464f-4668-aefa-dda94c758411]","process.thread.name":"kafka-producer-network-thread | producer-1","log.logger":"com.target.fulfillment.wmsodspublisher.kafka.KafkaPublisher"}

{"@timestamp":"2024-09-05T22:42:54.123Z","ecs.version":"1.2.0","log.level":"ERROR","message":"Mining session stopped due to error.","process.thread.name":"debezium-oracleconnector-ed51-scan3-3881-change-event-source-coordinator","log.logger":"io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource","dbz.connectorContext":"streaming","dbz.connectorName":"CONNECTORNAME","dbz.connectorType":"Oracle","dbz.databaseName":"DBNAME","dbz.taskId":"0","error.type":"java.sql.SQLException","error.message":"ORA-00600: internal error code, arguments: [krvrdccs30], [2048], [1], [4099], [1], [0], [], [], [], [], [], []\n","error.stack_trace":"java.sql.SQLException: ORA-00600: internal error code, arguments: [krvrdccs30], [2048], [1], [4099], [1], [0], [], [], [], [], [], []\n\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:630)\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:564)\n\tat oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1151)\n\tat oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:771)\n\tat oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:299)\n\tat oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:498)\n\tat oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:152)\n\tat oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:937)\n\tat oracle.jdbc.driver.OracleStatement.prepareDefineBufferAndExecute(OracleStatement.java:1172)\n\tat oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1101)\n\tat oracle.jdbc.driver.OracleStatement.executeSQLSelect(OracleStatement.java:1426)\n\tat oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1309)\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3746)\n\tat oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3855)\n\tat oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1098)\n\tat io.debezium.connector.oracle.logminer.processor.AbstractLogMinerEventProcessor.process(AbstractLogMinerEventProcessor.java:231)\n\tat io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:248)\n\tat io.debezium.connector.oracle.logminer.LogMinerStreamingChangeEventSource.execute(LogMinerStreamingChangeEventSource.java:62)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:271)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:194)\n\tat io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:137)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by: Error : 600, Position : 49, Sql = SELECT SCN, SQL_REDO, OPERATION_CODE, TIMESTAMP, XID, CSF, TABLE_NAME, SEG_OWNER, OPERATION, USERNAME, ROW_ID, ROLLBACK, RS_ID, STATUS, INFO, SSN, THREAD# FROM V$LOGMNR_CONTENTS WHERE SCN > :1  AND SCN <= :2  AND (((OPERATION_CODE IN (1,2,3,6,7,34,36,255)) AND (SEG_OWNER IS NULL OR SEG_OWNER NOT IN ('APPQOSSYS','AUDSYS','CTXSYS','DVSYS','DBSFWUSER','DBSNMP','GGSHAREDCAP','GSMADMIN_INTERNAL','LBACSYS','MDSYS','OJVMSYS','OLAPSYS','ORDDATA','ORDSYS','OUTLN','SYS','SYSTEM','WMSYS','XDB'))) OR (OPERATION_CODE = 5 AND INFO NOT LIKE 'INTERNAL DDL%')), OriginalSql = SELECT SCN, SQL_REDO, OPERATION_CODE, TIMESTAMP, XID, CSF, TABLE_NAME, SEG_OWNER, OPERATION, USERNAME, ROW_ID, ROLLBACK, RS_ID, STATUS, INFO, SSN, THREAD# FROM V$LOGMNR_CONTENTS WHERE SCN > ? AND SCN <= ? AND (((OPERATION_CODE IN (1,2,3,6,7,34,36,255)) AND (SEG_OWNER IS NULL OR SEG_OWNER NOT IN ('APPQOSSYS','AUDSYS','CTXSYS','DVSYS','DBSFWUSER','DBSNMP','GGSHAREDCAP','GSMADMIN_INTERNAL','LBACSYS','MDSYS','OJVMSYS','OLAPSYS','ORDDATA','ORDSYS','OUTLN','SYS','SYSTEM','WMSYS','XDB'))) OR (OPERATION_CODE = 5 AND INFO NOT LIKE 'INTERNAL DDL%')), Error Msg = ORA-00600: internal error code, arguments: [krvrdccs30], [2048], [1], [4099], [1], [0], [], [], [], [], [], []\n\n\tat oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:637)\n\t... 25 more\n"}

DB LOGS AR THE SAME TIME WHEN WE GOT THIS ERROR AND DEBEZIUM LISTENER APPLICATIONS STOPPED FOR EVER WITHOUT RECOVERING, UNLESS WE DO A CLEAN RESTART

2024-09-05T17:42:43.636153-05:00
*************************************************************************
Fatal NI connect error 12170.
  VERSION INFORMATION:
        TNS for Linux: Version 19.0.0.0.0 - Production
        Oracle Bequeath NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 19.0.0.0.0 - Production
  Version 19.23.0.0.0
  Time: 05-SEP-2024 17:42:43
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535
TNS-12535: TNS:operation timed out
    ns secondary err code: 12560
    nt main err code: 505
TNS-00505: Operation timed out
    nt secondary err code: 110
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=10.95.60.29)(PORT=36082))
2024-09-05T17:42:43.637019-05:00
NET  (PID:76125): Possible network disconnect with primary database