Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
1.5.0.Beta2
-
None
-
False
-
False
-
Undefined
Description
Hello!
I'am having problems ingesting the data (Downloading the snapshot is already working fast when i use it)
I'm using this configuration, to only test the "real time" modifications on the database.
{
"name": "kafka-rds-0004",
"config": {
"connector.class": "io.debezium.connector.oracle.OracleConnector",
"tasks.max": "1",
"database.server.name": "kafka-rds-0001",
"database.hostname": "server",
"database.port": "1521",
"database.user": "user",
"database.password": "pass",
"database.dbname": "dbname",
"database.history.kafka.bootstrap.servers": "http://kafka-confluent-cp-kafka.kafka-rds.svc.cluster.local:9092",
"database.history.kafka.topic": "schema-kafka-rds-0001",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://kafka-confluent-cp-schema-registry.kafka-rds.svc.cluster.local:8081",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://kafka-confluent-cp-schema-registry.kafka-rds.svc.cluster.local:8081",
"errors.log.enable": "true",
"database.connection.adapter": "logminer",
"log.mining.archive.log.hours": 24,
"snapshot.fetch.size": 2000,
"snapshot.mode": "schema_only",
"schema.include.list": "SCHEMA",
"table.include.list": "SCHEMA.TABLE,SCHEMA.TABLE_XA,SCHEMA.TABLE_2,SCHEMA.TABLE_3",
"column.include.list": "SCHEMA.TABLE.ROW_ID,SCHEMA.TABLE.BILL_ACCNT_ID,SCHEMA.TABLE.STATUS_CD,SCHEMA.TABLE.PROD_ID,SCHEMA.TABLE.ASSET_NUM,SCHEMA.TABLE.START_DT,SCHEMA.TABLE.END_DT,SCHEMA.TABLE_2.ROW_ID,SCHEMA.TABLE_XA.ROW_ID,SCHEMA.TABLE_XA.CHAR_VAL,SCHEMA.TABLE_XA.ASSET_ID,SCHEMA.TABLE_XA.ATTR_NAME,SCHEMA.TABLE_2.OU_SUM,SCHEMA.TABLE_2.OU_TYPE_CD,SCHEMA.TABLE_2.X_DUNN_LEVEL,SCHEMA.TABLE_2.CUST_STAT_CD,SCHEMA.TABLE_3.CATEGORY_CD,SCHEMA.TABLE_3.SUB_TYPE_CD,SCHEMA.TABLE_3.ROW_ID,SCHEMA.TABLE_3.NAME",
"transforms": "unwrap",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"transforms.unwrap.drop.tombstones": "false",
"transforms.unwrap.delete.handling.mode": "rewrite"
}
}
I updated one register on the Oracle database and inserted one as well.
Both at 13:52. But only the update was reflected on the kafka connect at 14:02
In previous tests i have already got 20 minutes of delay. It's taking too long right? Or this performance is expected?
Bellow are the logs from the connector.
1-03-19 13:50:27,848] INFO Context created (io.debezium.pipeline.ChangeEventSourceCoordinator) │ │ [2021-03-19 13:50:27,848] INFO Snapshot ended with SnapshotResult [status=SKIPPED, offset=OracleOffsetContext [scn=760115763]] (io.debezium.pipeline.ChangeEventSourceCoordinator) │ │ [2021-03-19 13:50:27,849] INFO Connected metrics set to 'true' (io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics) │ │ [2021-03-19 13:50:27,849] INFO Starting streaming (io.debezium.pipeline.ChangeEventSourceCoordinator) │ │ [2021-03-19 13:50:27,849] INFO Logminer metrics initialized LogMinerMetrics{currentScn=-1, logMinerQueryCount=0, totalProcessedRows=0, totalCapturedDmlCount=0, totalDurationOfFetchingQuery=PT0S, lastCapturedDmlCoun │ │ [2021-03-19 13:51:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:51:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:52:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:52:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:53:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:53:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:54:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:54:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:55:10,100] INFO [Worker clientId=connect-1, groupId=kafka-confluent] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder) │ │ [2021-03-19 13:55:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:55:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:56:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:56:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:57:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:57:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:58:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:58:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:59:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 13:59:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:00:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:00:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:01:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:01:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:02:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:02:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:03:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:03:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:03:52,348] INFO 1 records sent during previous 00:13:28.173, last recorded offset: {commit_scn=760220398, transaction_id=null, scn=760220396} (io.debezium.connector.common.BaseSourceTask) │ │ [2021-03-19 14:04:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:04:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:04:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Finished commitOffsets successfully in 6 ms (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:05:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:05:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:06:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) │ │ [2021-03-19 14:06:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
