Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5557

Supply partition when comiting offsets with source database

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 2.0.0.Beta2
    • 1.9.5.Final
    • core-library
    • None
    • False
    • None
    • False

      Which use case/requirement will be addressed by the proposed feature?

      We'd like to ease the deletion of Capture Tables on the SQL Server database side by letting it know that we've already consumed all records from a Capture Table and it can be safely deleted. For that we'd like to use io.debezium.pipeline.source.spi.StreamingChangeEventSource#commitOffset(Map<String, ?> offset) which tells the offset that was committed with Kafka Connect. The problem is that there is no information on which partition the offset being committed belongs to.

      Implementation ideas (optional)

      It makes sense to change the signature of commitOffset to (Map<String, ?> partition, Map<String, ?> offset) so that the implementation of this method knows which partition the offset belongs to. For that to work io.debezium.connector.common.BaseSourceTask needs to track not only the last offset that was consumed, but also the partition that last record was consumed from.

      https://github.com/debezium/debezium/pull/3828

              Unassigned Unassigned
              vramanenka@sugarcrm.com Vadzim Ramanenka (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: