Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5327

Smart Backfills | Ability to backfill selective data

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 2.0.0.Beta2
    • None
    • mysql-connector
    • None

      At present, when a user triggers an incremental backfill for a table, Debezium backfills the whole table. But what if a user wants to backfill a subset of table's data. 

      For example, let's assume there is a products table. And a user wants to backfill only the records of the products table which have product_id=X. 
      Atm, the query that is executed behind the scenes, for the backfill,  is like

      SELECT * FROM products .........

      I'd like to extend the existing incremental backfill feature to accept a new user param, additional-condition. Using additional-condition, a user will be able to specify additional context for the db query. 

      Current kafka incremental backfill signal:

      {"type":"execute-snapshot", "data": {""data-collections":["some_db.products"], "type":"INCREMENTAL"}}

      Proposed kafka incremental backfill signal:

      {"type":"execute-snapshot", "data": {""data-collections":["some_db.products"], "type":"INCREMENTAL", "additional-condition":"product_id=X"}}

      This will modify the query executed for the backfill to something like:
      SELECT * FROM products WHERE product_id=X .........

       

            Unassigned Unassigned
            vivekwassan Vivek Wassan (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: