Details
-
Feature Request
-
Resolution: Done
-
Major
-
0.10.0.Final
-
None
Description
After starting SQL Server CDC, the data obtained is only the data as of the moment of startup. The real-time changed records are not obtained, and it just can be retrieved after restarting.
Finally, you have to restart constantly to get the data.
My Configuration:
name=SqlServerConnector
connector.class=io.debezium.connector.sqlserver.SqlServerConnector
tasks.max=1
database.hostname=xxx
database.port=1433
database.dbname=PCA
database.user=xxx
database.password=xxx
database.server.name=pca3
table.whitelist=dbo.PCA_LOG
database.history.kafka.bootstrap.servers=xxx
database.history.kafka.topic=dbhistory.tao.pcalog3
snapshot.mode=initial_schema_only
snapshot.isolation.mode=snapshot
transforms=unwrap
transforms.unwrap.drop.tombstones=false
transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
Kafka Connect Log:
[2019-11-30 08:06:38,866] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2019-11-30 08:06:41,874] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2019-11-30 08:06:41,874] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Setting newly assigned partitions: dbhistory.tao.pcalog3-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2019-11-30 08:06:41,878] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Found no committed offset for partition dbhistory.tao.pcalog3-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator) [2019-11-30 08:06:41,881] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Resetting offset for partition dbhistory.tao.pcalog3-0 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState) [2019-11-30 08:06:41,886] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Member pca3-dbhistory-ff0e40be-b6ef-4800-bb0d-9b084de49af0 sending LeaveGroup request to coordinator 10.99.169.213:9092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator) [2019-11-30 08:06:41,891] INFO Requested thread factory for connector SqlServerConnector, id = pca3 named = error-handler (io.debezium.util.Threads) [2019-11-30 08:06:41,891] INFO Requested thread factory for connector SqlServerConnector, id = pca3 named = change-event-source-coordinator (io.debezium.util.Threads) [2019-11-30 08:06:41,891] INFO Creating thread debezium-sqlserverconnector-pca3-change-event-source-coordinator (io.debezium.util.Threads) [2019-11-30 08:06:41,891] INFO WorkerSourceTask{id=SqlServerConnector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:41,891] INFO A previous offset indicating a completed snapshot has been found. Neither schema nor data will be snapshotted. (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource) [2019-11-30 08:06:41,954] INFO Last position recorded in offsets is 00014563:00006f78:0007(00014563:00006f78:0002)[1] (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource) [2019-11-30 08:06:42,180] INFO Skipping change ChangeTablePointer [changeTable=Capture instance "dbo_PCA_LOG" [sourceTableId=PCA.dbo.PCA_LOG, changeTableId=PCA.cdc.dbo_PCA_LOG_CT, startLsn=00014533:00021188:0010, changeTableObjectId=200088199, stopLsn=NULL], resultSet=SQLServerResultSet:19319, completed=false, currentChangePosition=00014563:00006f78:0007(00014563:00006f78:0002)] as its order in the transaction 1 is smaller than or equal to the last recorded operation 00014563:00006f78:0007(00014563:00006f78:0002)[1] (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource) [2019-11-30 08:06:47,444] INFO WorkerSourceTask{id=SqlServerConnector_readonly-Demo-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:47,444] INFO WorkerSourceTask{id=SqlServerConnector_readonly-Demo-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:47,823] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:47,823] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:47,829] INFO WorkerSourceTask{id=SqlServerConnector-0} Finished commitOffsets successfully in 6 ms (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:57,829] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:57,829] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:06:57,833] INFO WorkerSourceTask{id=SqlServerConnector-0} Finished commitOffsets successfully in 4 ms (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:07:07,833] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [2019-11-30 08:07:07,833] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)