Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9614

Oracle initial snapshot takes too long to complete

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • None
    • oracle-connector
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

       
      I have a table with a total of 20 million rows of data, and the total data size is 6272MB. Debezium and Kafka are deployed on the same server, which has a configuration of 16 cores, 32GB RAM, and 500GB SSD. The Oracle database is located on another server in the local area network, and this server also has a configuration of 16 cores, 32GB RAM, and 500GB SSD. I perform snapshots via the Oracle connector, and after multiple tests, I found that the entire snapshot phase takes approximately 15 minutes. No matter how I adjust the values of any parameters, I cannot improve the snapshot speed.
      The parameters I have tried modifying include:

      max.queue.size,
      snapshot.delay.ms,
      poll.interval.ms,
      snapshot.fetch.size,
      max.batch.size,
      topic.creation.default.partitions,
      topic.creation.default.compression.type,
      producer.override.batch.size,
      producer.override.linger.ms,
      producer.override.max.request.size,
      producer.override.send.buffer.bytes.

      The logs show that the number of exported data increases by 200,000 to 300,000 rows every 10 seconds. Could you please tell me how to improve the snapshot speed, preferably completing the snapshot of the entire table within 5 minutes?

      What Debezium connector do you use and what version?

      debezium docker version: 3.0.0.Final

      What is the connector configuration?

      {
          "name": "oracle1332pg199",
          "config":{        
                  "connector.class": "io.debezium.connector.oracle.OracleConnector",        
                  "max.queue.size": "50000",        
                  "tasks.max": "1",        
                 "database.query.timeout.ms": "1200000",        
                 "schema.include.list": "CDCUSER",        
                 "snapshot.delay.ms": "0",        
                 "topic.prefix": "oracle1332pg199",        
                 "heartbeat.action.query": "select 1 from dual",        
                 "schema.history.internal.kafka.topic": "oracle1332pg199_SCHEMA_CHANGES",        
                  "poll.interval.ms": "100",        
                  "snapshot.fetch.size": "20000",        
                  "database.user": "*****",        
                  "database.dbname": "CDCDUSER",        
                  "time.precision.mode": "connect",        
                  "schema.history.internal.kafka.bootstrap.servers": "192.168.5.131:9092,192.168.5.137:9092",                                           
                   "heartbeat.interval.ms": "60000",        
                   "snapshot.max.threads": "1",        
                   "database.port": "1521",      
                   "errors.max.retries": "1",        
                   "database.hostname": "192.168.5.136",        
                   "database.password": "******",        
                   "name": "oracle1332pg199",        
                   "table.include.list": "CDCUSER.TEST_TABLE_0009",        
                   "max.batch.size": "20000",        
                   "snapshot.mode": "initial",        
                   "database.url":"jdbc:oracle:thin:@//192.168.5.136:1521/ORCL",        
                   "topic.creation.default.replication.factor": 1,        
                   "topic.creation.default.partitions": 20,        
                   "topic.creation.default.compression.type": "lz4",        
                   "producer.override.acks":"1",        
                   "producer.override.batch.size": "1048576",        
                   "producer.override.linger.ms": "10",        
                   "producer.override.max.request.size": "16777216",        
                   "producer.override.send.buffer.bytes": "16777216",        
                   "producer.override.buffer.memory":"268435456",        
                   "producer.override.compression.type": "lz4",        
                   "producer.override.compression.lz4.level": "1",        
                   "producer.override.max.in.flight.requests.per.connection":"5",        
                   "log.mining.scn.gap.detection.time.interval.max.ms":"100000"    
      }
      

      What is the captured database version and mode of deployment?

      Oracle version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

      What behavior do you expect?

      I want to be able to control the snapshot speed.

      What behavior do you see?

      I cannot change the snapshot speed by modifying any snapshot-related parameters; it remains consistently at approximately 20,000-30,000 rows per second.

      Do you see the same behaviour using the latest released Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      No, just used debezium 3.0.0.Final

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      This is a part of logs:

      2025-10-24 06:13:49,025 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 2462018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:01:50.377
      2025-10-24 06:13:51,808 - INFO  [qtp1856277665-65:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:13:51 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 6
      2025-10-24 06:13:57,070 - INFO  [SourceTaskOffsetCommitter-1:WorkerSourceTask@235] - WorkerSourceTask{id=oracle1332pg198-0} Committing offsets for 1299999 acknowledged messages
      2025-10-24 06:13:59,027 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 2686786 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:00.379
      2025-10-24 06:14:01,874 - INFO  [qtp1856277665-66:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:01 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
      2025-10-24 06:14:09,433 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 2902018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:10.784
      2025-10-24 06:14:11,964 - INFO  [qtp1856277665-140:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:11 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
      2025-10-24 06:14:19,454 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 3102018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:20.806
      2025-10-24 06:14:22,029 - INFO  [qtp1856277665-63:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:22 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
      2025-10-24 06:14:29,634 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 3322018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:30.985
      2025-10-24 06:14:32,110 - INFO  [qtp1856277665-68:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:32 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 7
      2025-10-24 06:14:32,316 - INFO  [task-thread-oracle1332pg198-0:BaseSourceTask@349] - 1720000 records sent during previous 00:01:19.918, last recorded offset of {server=oracle1332pg198} partition is {snapshot_scn=7944761, snapshot=true, scn=7944761, snapshot_completed=false}
      2025-10-24 06:14:39,646 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 3545436 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:40.998
      2025-10-24 06:14:42,197 - INFO  [qtp1856277665-67:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:42 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 6
      2025-10-24 06:14:49,816 - INFO  [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] -      Exported 3762018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:51.167
      2025-10-24 06:14:52,279 - INFO  [qtp1856277665-79:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:52 +0000] "GET / HTTP/1.1"

       

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              jjdyr23@163.com shuren zhou
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: