Loading...

Type: Bug
Resolution: Not a Bug
Priority: Major
Fix Version/s: None
Affects Version/s: 1.5.0.Beta2
Component/s: oracle-connector
Labels:
None

Blocked:
False
Ready:
False
Release Note Text:
Undefined
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Hello!

I'am having problems ingesting the data (Downloading the snapshot is already working fast when i use it)

I'm using this configuration, to only test the "real time" modifications on the database.

{
    "name": "kafka-rds-0004",
    "config": {
        "connector.class": "io.debezium.connector.oracle.OracleConnector",
        "tasks.max": "1",
        "database.server.name": "kafka-rds-0001",
        "database.hostname": "server",
        "database.port": "1521",
        "database.user": "user",
        "database.password": "pass",
        "database.dbname": "dbname",
        "database.history.kafka.bootstrap.servers": "http://kafka-confluent-cp-kafka.kafka-rds.svc.cluster.local:9092",
        "database.history.kafka.topic": "schema-kafka-rds-0001",
        "key.converter": "io.confluent.connect.avro.AvroConverter",
        "key.converter.schema.registry.url": "http://kafka-confluent-cp-schema-registry.kafka-rds.svc.cluster.local:8081",
        "value.converter": "io.confluent.connect.avro.AvroConverter",
        "value.converter.schema.registry.url": "http://kafka-confluent-cp-schema-registry.kafka-rds.svc.cluster.local:8081",
        "errors.log.enable": "true",
        "database.connection.adapter": "logminer",
        "log.mining.archive.log.hours": 24,

        "snapshot.fetch.size": 2000,

        "snapshot.mode": "schema_only",

        "schema.include.list": "SCHEMA",
        "table.include.list": "SCHEMA.TABLE,SCHEMA.TABLE_XA,SCHEMA.TABLE_2,SCHEMA.TABLE_3",
        "column.include.list": "SCHEMA.TABLE.ROW_ID,SCHEMA.TABLE.BILL_ACCNT_ID,SCHEMA.TABLE.STATUS_CD,SCHEMA.TABLE.PROD_ID,SCHEMA.TABLE.ASSET_NUM,SCHEMA.TABLE.START_DT,SCHEMA.TABLE.END_DT,SCHEMA.TABLE_2.ROW_ID,SCHEMA.TABLE_XA.ROW_ID,SCHEMA.TABLE_XA.CHAR_VAL,SCHEMA.TABLE_XA.ASSET_ID,SCHEMA.TABLE_XA.ATTR_NAME,SCHEMA.TABLE_2.OU_SUM,SCHEMA.TABLE_2.OU_TYPE_CD,SCHEMA.TABLE_2.X_DUNN_LEVEL,SCHEMA.TABLE_2.CUST_STAT_CD,SCHEMA.TABLE_3.CATEGORY_CD,SCHEMA.TABLE_3.SUB_TYPE_CD,SCHEMA.TABLE_3.ROW_ID,SCHEMA.TABLE_3.NAME",

        "transforms": "unwrap",

        "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
        "transforms.unwrap.drop.tombstones": "false",
        "transforms.unwrap.delete.handling.mode": "rewrite"

    }
}

I updated one register on the Oracle database and inserted one as well.

Both at 13:52. But only the update was reflected on the kafka connect at 14:02

In previous tests i have already got 20 minutes of delay. It's taking too long right? Or this performance is expected?

Bellow are the logs from the connector.

1-03-19 13:50:27,848] INFO Context created (io.debezium.pipeline.ChangeEventSourceCoordinator)                                                                                                                     │
│ [2021-03-19 13:50:27,848] INFO Snapshot ended with SnapshotResult [status=SKIPPED, offset=OracleOffsetContext [scn=760115763]] (io.debezium.pipeline.ChangeEventSourceCoordinator)                                     │
│ [2021-03-19 13:50:27,849] INFO Connected metrics set to 'true' (io.debezium.pipeline.metrics.StreamingChangeEventSourceMetrics)                                                                                        │
│ [2021-03-19 13:50:27,849] INFO Starting streaming (io.debezium.pipeline.ChangeEventSourceCoordinator)                                                                                                                  │
│ [2021-03-19 13:50:27,849] INFO Logminer metrics initialized LogMinerMetrics{currentScn=-1, logMinerQueryCount=0, totalProcessedRows=0, totalCapturedDmlCount=0, totalDurationOfFetchingQuery=PT0S, lastCapturedDmlCoun │
│ [2021-03-19 13:51:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:51:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:52:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:52:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:53:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:53:24,180] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:54:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:54:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:55:10,100] INFO [Worker clientId=connect-1, groupId=kafka-confluent] Session key updated (org.apache.kafka.connect.runtime.distributed.DistributedHerder)                                               │
│ [2021-03-19 13:55:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:55:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:56:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:56:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:57:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:57:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:58:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:58:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 13:59:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 13:59:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:00:24,181] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:00:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:01:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:01:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:02:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:02:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:03:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:03:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:03:52,348] INFO 1 records sent during previous 00:13:28.173, last recorded offset: {commit_scn=760220398, transaction_id=null, scn=760220396} (io.debezium.connector.common.BaseSourceTask)             │
│ [2021-03-19 14:04:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:04:24,182] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:04:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Finished commitOffsets successfully in 6 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                   │
│ [2021-03-19 14:05:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:05:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)                                             │
│ [2021-03-19 14:06:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)                                                                            │
│ [2021-03-19 14:06:24,188] INFO WorkerSourceTask{id=kafka-rds-0004-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates