-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
None
-
None
-
False
-
-
False
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
I have a table with a total of 20 million rows of data, and the total data size is 6272MB. Debezium and Kafka are deployed on the same server, which has a configuration of 16 cores, 32GB RAM, and 500GB SSD. The Oracle database is located on another server in the local area network, and this server also has a configuration of 16 cores, 32GB RAM, and 500GB SSD. I perform snapshots via the Oracle connector, and after multiple tests, I found that the entire snapshot phase takes approximately 15 minutes. No matter how I adjust the values of any parameters, I cannot improve the snapshot speed.
The parameters I have tried modifying include:
max.queue.size,
snapshot.delay.ms,
poll.interval.ms,
snapshot.fetch.size,
max.batch.size,
topic.creation.default.partitions,
topic.creation.default.compression.type,
producer.override.batch.size,
producer.override.linger.ms,
producer.override.max.request.size,
producer.override.send.buffer.bytes.
The logs show that the number of exported data increases by 200,000 to 300,000 rows every 10 seconds. Could you please tell me how to improve the snapshot speed, preferably completing the snapshot of the entire table within 5 minutes?
What Debezium connector do you use and what version?
debezium docker version: 3.0.0.Final
What is the connector configuration?
{
"name": "oracle1332pg199",
"config":{
"connector.class": "io.debezium.connector.oracle.OracleConnector",
"max.queue.size": "50000",
"tasks.max": "1",
"database.query.timeout.ms": "1200000",
"schema.include.list": "CDCUSER",
"snapshot.delay.ms": "0",
"topic.prefix": "oracle1332pg199",
"heartbeat.action.query": "select 1 from dual",
"schema.history.internal.kafka.topic": "oracle1332pg199_SCHEMA_CHANGES",
"poll.interval.ms": "100",
"snapshot.fetch.size": "20000",
"database.user": "*****",
"database.dbname": "CDCDUSER",
"time.precision.mode": "connect",
"schema.history.internal.kafka.bootstrap.servers": "192.168.5.131:9092,192.168.5.137:9092",
"heartbeat.interval.ms": "60000",
"snapshot.max.threads": "1",
"database.port": "1521",
"errors.max.retries": "1",
"database.hostname": "192.168.5.136",
"database.password": "******",
"name": "oracle1332pg199",
"table.include.list": "CDCUSER.TEST_TABLE_0009",
"max.batch.size": "20000",
"snapshot.mode": "initial",
"database.url":"jdbc:oracle:thin:@//192.168.5.136:1521/ORCL",
"topic.creation.default.replication.factor": 1,
"topic.creation.default.partitions": 20,
"topic.creation.default.compression.type": "lz4",
"producer.override.acks":"1",
"producer.override.batch.size": "1048576",
"producer.override.linger.ms": "10",
"producer.override.max.request.size": "16777216",
"producer.override.send.buffer.bytes": "16777216",
"producer.override.buffer.memory":"268435456",
"producer.override.compression.type": "lz4",
"producer.override.compression.lz4.level": "1",
"producer.override.max.in.flight.requests.per.connection":"5",
"log.mining.scn.gap.detection.time.interval.max.ms":"100000"
}
What is the captured database version and mode of deployment?
Oracle version: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
What behavior do you expect?
I want to be able to control the snapshot speed.
What behavior do you see?
I cannot change the snapshot speed by modifying any snapshot-related parameters; it remains consistently at approximately 20,000-30,000 rows per second.
Do you see the same behaviour using the latest released Debezium version?
(Ideally, also verify with latest Alpha/Beta/CR version)
No, just used debezium 3.0.0.Final
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
This is a part of logs:
2025-10-24 06:13:49,025 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 2462018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:01:50.377
2025-10-24 06:13:51,808 - INFO [qtp1856277665-65:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:13:51 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 6
2025-10-24 06:13:57,070 - INFO [SourceTaskOffsetCommitter-1:WorkerSourceTask@235] - WorkerSourceTask{id=oracle1332pg198-0} Committing offsets for 1299999 acknowledged messages
2025-10-24 06:13:59,027 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 2686786 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:00.379
2025-10-24 06:14:01,874 - INFO [qtp1856277665-66:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:01 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
2025-10-24 06:14:09,433 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 2902018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:10.784
2025-10-24 06:14:11,964 - INFO [qtp1856277665-140:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:11 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
2025-10-24 06:14:19,454 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 3102018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:20.806
2025-10-24 06:14:22,029 - INFO [qtp1856277665-63:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:22 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 5
2025-10-24 06:14:29,634 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 3322018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:30.985
2025-10-24 06:14:32,110 - INFO [qtp1856277665-68:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:32 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 7
2025-10-24 06:14:32,316 - INFO [task-thread-oracle1332pg198-0:BaseSourceTask@349] - 1720000 records sent during previous 00:01:19.918, last recorded offset of {server=oracle1332pg198} partition is {snapshot_scn=7944761, snapshot=true, scn=7944761, snapshot_completed=false}
2025-10-24 06:14:39,646 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 3545436 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:40.998
2025-10-24 06:14:42,197 - INFO [qtp1856277665-67:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:42 +0000] "GET / HTTP/1.1" 200 91 "-" "curl/8.6.0" 6
2025-10-24 06:14:49,816 - INFO [pool-21-thread-1:RelationalSnapshotChangeEventSource@642] - Exported 3762018 records for table 'CDCDUSER.CDCUSER.TEST_TABLE_0009' after 00:02:51.167
2025-10-24 06:14:52,279 - INFO [qtp1856277665-79:Slf4jRequestLogWriter@62] - [0:0:0:0:0:0:0:1] - - [24/Oct/2025:06:14:52 +0000] "GET / HTTP/1.1"
How to reproduce the issue using our tutorial deployment?
<Your answer>
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
<Your answer>
Implementation ideas (optional)
<Your answer>