-
Enhancement
-
Resolution: Done
-
Major
-
None
-
None
I use Debezium MSSQL to get data from ProductLogs and then push to kafka
I found that the Metric MillisecondsBehindSource is high. The delay increase when the running time increase (60s --> 300s)
The delay showed in attached file.
I changed the BatchSize to 4096, PullInterval to 200ms but it's effective.
Speed of data in ProductLogs arround 10/seconds.
The size of data arround 400KB-1024KB
---------
Below is my configuration
connector.class=io.debezium.connector.sqlserver.SqlServerConnector
database.dbname=xxx_PRODUCT
database.user=sapo_dev_kafka_connect
transforms.unwrap.delete.handling.mode=rewrite
transforms.insertKey.fields=TenantId
transforms.extractKey.field=TenantId
tasks.max=1
transforms.insertKey.type=org.apache.kafka.connect.transforms.ValueToKey
database.history.kafka.bootstrap.servers=xxx
database.history.kafka.topic=sqlserver.dbhistory.xxx
transforms=unwrap,insertKey,extractKey
database.server.name=xxx
transforms.extractKey.type=org.apache.kafka.connect.transforms.ExtractField$Key
database.port=1433
table.whitelist=dbo.ProductLogs,dbo.Categories,dbo.ProductLogs
database.hostname=xxx
database.password=xxx
transforms.unwrap.drop.tombstones=true
transforms.unwrap.type=io.debezium.transforms.UnwrapFromEnvelope
transforms.unwrap.operation.header=true
snapshot.mode=initial_schema_only