Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4997

Oracle Connector produces duplicated records when restoration

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Critical Critical
    • None
    • 1.6.4.Final
    • oracle-connector
    • None
    • False
    • None
    • False

      What Debezium connector do you use and what version?

      oracle connector, version: 1.6.4.Final

      What is the connector configuration?

      74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask - Starting OracleConnectorTask with configuration:74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    connector.class = io.debezium.connector.oracle.OracleConnector74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.user = system74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.dbname = XE74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    offset.storage = com.ververica.cdc.debezium.internal.FlinkOffsetBackingStore74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.server.name = oracle_logminer74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    include.schema.changes = false74910 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.port = 5512974911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    offset.flush.interval.ms = 922337203685477580774911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    tombstones.on.delete = false74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.hostname = localhost74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.password = ********74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    offset.storage.flink.state.value = {"sourcePartition":{"server":"oracle_logminer"},"sourceOffset":{"snapshot":true,"scn":"397461","snapshot_completed":true}}74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    name = engine74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.history.skip.unparseable.ddl = true74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.history.store.only.monitored.tables.ddl = false74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    table.include.list = debezium.products74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.history.instance.name = 3190eed3-cf89-4336-9269-bd2778c3ebb174911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    snapshot.mode = initial74911 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask -    database.history = com.ververica.cdc.debezium.internal.FlinkDatabaseSchemaHistory74963 [debezium-engine] INFO  io.debezium.connector.oracle.OracleConnection - Database Version: Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production 

      What is the captured database version and mode of depoyment?

       
      Oracle Database: 11g Express Edition

      Do you see the same behaviour using the latest relesead Debezium version?

      1.5.4.Final works fine, when I bump debezium version, the bug appears....

      Do you have the connector logs, ideally from start till finish?

      The record with scn=407624 appears again after restoration, I try to find debezium code and found nothing.

      74979 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask - Found previous offset OracleOffsetContext [scn=397461]
      74982 [debezium-oracleconnector-oracle_logminer-change-event-source-coordinator] INFO  io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming
      178011 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask - 1 records sent during previous 00:01:43.073, last recorded offset: {commit_scn=408721, transaction_id=null, scn=407624}
      Record @ (undef) : SourceRecord{sourcePartition={server=oracle_logminer}, sourceOffset={commit_scn=408721, transaction_id=null, scn=407624}} ConnectRecord{topic='oracle_logminer.DEBEZIUM.PRODUCTS', kafkaPartition=null, key=Struct{ID=110}, keySchema=Schema{oracle_logminer.DEBEZIUM.PRODUCTS.Key:STRUCT}, value=Struct{after=Struct{ID=110,NAME=robot,DESCRIPTION=Toy robot,WEIGHT=Struct{scale=3,value=[B@2503ec73}},source=Struct{version=1.6.4.Final,connector=oracle,name=oracle_logminer,ts_ms=1650011591000,db=XE,schema=DEBEZIUM,table=PRODUCTS,txId=08001f0093010000,scn=407624,commit_scn=408721},op=c,ts_ms=1650011689791}, valueSchema=Schema{oracle_logminer.DEBEZIUM.PRODUCTS.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
      
      restore here
      
      180386 [debezium-oracleconnector-oracle_logminer-change-event-source-coordinator] INFO  io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming
      232885 [debezium-engine] INFO  io.debezium.connector.common.BaseSourceTask - 1 records sent during previous 00:00:52.574, last recorded offset: {commit_scn=408721, transaction_id=null, scn=407624}
      Record @ (undef) : SourceRecord{sourcePartition={server=oracle_logminer}, sourceOffset={commit_scn=408721, transaction_id=null, scn=407624}} ConnectRecord{topic='oracle_logminer.DEBEZIUM.PRODUCTS', kafkaPartition=null, key=Struct{ID=110}, keySchema=Schema{oracle_logminer.DEBEZIUM.PRODUCTS.Key:STRUCT}, value=Struct{after=Struct{ID=110,NAME=robot,DESCRIPTION=Toy robot,WEIGHT=Struct{scale=3,value=[B@285c6918}},source=Struct{version=1.6.4.Final,connector=oracle,name=oracle_logminer,ts_ms=1650011591000,db=XE,schema=DEBEZIUM,table=PRODUCTS,txId=08001f0093010000,scn=407624,commit_scn=408721},op=c,ts_ms=1650011744629}, valueSchema=Schema{oracle_logminer.DEBEZIUM.PRODUCTS.Envelope:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}
         

      How to reproduce the issue using our tutorial deployment?

      Run unit test OracleSourceTest#testCheckpointAndRestore() in this PR

      https://github.com/ververica/flink-cdc-connectors/pull/992/files#diff-4a725f60d2533332b44052b49a5129bdeac4764e60182859a6177622c651321e

              ccranfor@redhat.com Chris Cranford
              leonardxu leonard xu (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: