Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1642

Support streaming changes from SQL Server "AlwaysOn" replica

    XMLWordPrintable

Details

    Description

      After starting SQL Server CDC, the data obtained is only the data as of the moment of startup. The real-time changed records are not obtained, and it just can be retrieved after restarting.

      Finally, you have to restart constantly to get the data.

      My Configuration:

      name=SqlServerConnector
      connector.class=io.debezium.connector.sqlserver.SqlServerConnector
      tasks.max=1
      database.hostname=xxx
      database.port=1433
      database.dbname=PCA
      database.user=xxx
      database.password=xxx
      database.server.name=pca3
      table.whitelist=dbo.PCA_LOG
      database.history.kafka.bootstrap.servers=xxx
      database.history.kafka.topic=dbhistory.tao.pcalog3
      snapshot.mode=initial_schema_only
      snapshot.isolation.mode=snapshot
      transforms=unwrap
      transforms.unwrap.drop.tombstones=false
      transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState
      

      Kafka Connect Log:

      [2019-11-30 08:06:38,866] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      [2019-11-30 08:06:41,874] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      [2019-11-30 08:06:41,874] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Setting newly assigned partitions: dbhistory.tao.pcalog3-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
      [2019-11-30 08:06:41,878] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Found no committed offset for partition dbhistory.tao.pcalog3-0 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
      [2019-11-30 08:06:41,881] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Resetting offset for partition dbhistory.tao.pcalog3-0 to offset 0. (org.apache.kafka.clients.consumer.internals.SubscriptionState)
      [2019-11-30 08:06:41,886] INFO [Consumer clientId=pca3-dbhistory, groupId=pca3-dbhistory] Member pca3-dbhistory-ff0e40be-b6ef-4800-bb0d-9b084de49af0 sending LeaveGroup request to coordinator 10.99.169.213:9092 (id: 2147483646 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
      [2019-11-30 08:06:41,891] INFO Requested thread factory for connector SqlServerConnector, id = pca3 named = error-handler (io.debezium.util.Threads)
      [2019-11-30 08:06:41,891] INFO Requested thread factory for connector SqlServerConnector, id = pca3 named = change-event-source-coordinator (io.debezium.util.Threads)
      [2019-11-30 08:06:41,891] INFO Creating thread debezium-sqlserverconnector-pca3-change-event-source-coordinator (io.debezium.util.Threads)
      [2019-11-30 08:06:41,891] INFO WorkerSourceTask{id=SqlServerConnector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:41,891] INFO A previous offset indicating a completed snapshot has been found. Neither schema nor data will be snapshotted. (io.debezium.connector.sqlserver.SqlServerSnapshotChangeEventSource)
      [2019-11-30 08:06:41,954] INFO Last position recorded in offsets is 00014563:00006f78:0007(00014563:00006f78:0002)[1] (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource)
      [2019-11-30 08:06:42,180] INFO Skipping change ChangeTablePointer [changeTable=Capture instance "dbo_PCA_LOG" [sourceTableId=PCA.dbo.PCA_LOG, changeTableId=PCA.cdc.dbo_PCA_LOG_CT, startLsn=00014533:00021188:0010, changeTableObjectId=200088199, stopLsn=NULL], resultSet=SQLServerResultSet:19319, completed=false, currentChangePosition=00014563:00006f78:0007(00014563:00006f78:0002)] as its order in the transaction 1 is smaller than or equal to the last recorded operation 00014563:00006f78:0007(00014563:00006f78:0002)[1] (io.debezium.connector.sqlserver.SqlServerStreamingChangeEventSource)
      [2019-11-30 08:06:47,444] INFO WorkerSourceTask{id=SqlServerConnector_readonly-Demo-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:47,444] INFO WorkerSourceTask{id=SqlServerConnector_readonly-Demo-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:47,823] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:47,823] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:47,829] INFO WorkerSourceTask{id=SqlServerConnector-0} Finished commitOffsets successfully in 6 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:57,829] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:57,829] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:06:57,833] INFO WorkerSourceTask{id=SqlServerConnector-0} Finished commitOffsets successfully in 4 ms (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:07:07,833] INFO WorkerSourceTask{id=SqlServerConnector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2019-11-30 08:07:07,833] INFO WorkerSourceTask{id=SqlServerConnector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            cobolbaby xing-long zhang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: