-
Enhancement
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
None
-
False
At present, when a user triggers an incremental backfill for a table, Debezium backfills the whole table. But what if a user wants to backfill a subset of table's data.
For example, let's assume there is a products table. And a user wants to backfill only the records of the products table which have product_id=X.
Atm, the query that is executed behind the scenes, for the backfill, is like
SELECT * FROM products .........
I'd like to extend the existing incremental backfill feature to accept a new user param, additional-condition. Using additional-condition, a user will be able to specify additional context for the db query.
Current kafka incremental backfill signal:
{"type":"execute-snapshot", "data": {""data-collections":["some_db.products"], "type":"INCREMENTAL"}}
Proposed kafka incremental backfill signal:
{"type":"execute-snapshot", "data": {""data-collections":["some_db.products"], "type":"INCREMENTAL", "additional-condition":"product_id=X"}}
This will modify the query executed for the backfill to something like:
SELECT * FROM products WHERE product_id=X .........