Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5557

Supply partition when comiting offsets with source database

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Done
    • Major
    • 2.0.0.Beta2
    • 1.9.5.Final
    • core-library
    • None
    • False
    • None
    • False
    • 0
    • 0% 0%

    Description

      Which use case/requirement will be addressed by the proposed feature?

      We'd like to ease the deletion of Capture Tables on the SQL Server database side by letting it know that we've already consumed all records from a Capture Table and it can be safely deleted. For that we'd like to use io.debezium.pipeline.source.spi.StreamingChangeEventSource#commitOffset(Map<String, ?> offset) which tells the offset that was committed with Kafka Connect. The problem is that there is no information on which partition the offset being committed belongs to.

      Implementation ideas (optional)

      It makes sense to change the signature of commitOffset to (Map<String, ?> partition, Map<String, ?> offset) so that the implementation of this method knows which partition the offset belongs to. For that to work io.debezium.connector.common.BaseSourceTask needs to track not only the last offset that was consumed, but also the partition that last record was consumed from.

      https://github.com/debezium/debezium/pull/3828

      Attachments

        Activity

          People

            Unassigned Unassigned
            vramanenka@sugarcrm.com Vadzim Ramanenka (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: