Details
-
Task
-
Resolution: Unresolved
-
Major
-
1.7.0.CR1
-
None
-
False
-
False
-
Undefined
Description
Currently, for each table with a custom snapshot select, one property with the name snapshot.select.statement.overrides.<table> needs to be given. This causes problems when configuring Debezium Server via environment variables, where each underscore in the key is replaced with a dot (see DBZ-3760). For tables that actually have an underscore in their name (e.g. "products_at_hand") this causes the snapshot select override to not be found.
Instead of giving one property per table, making the property key a variable part, all overrides should be given in a single property. The format should be the same as the one used for message.key.columns.
We might retrofit snapshot.select.statement.overrides accordingly via some heuristics, but there's a relation to the new notion of incremental snapshot overrides which we should consider: currently, snapshot overrides are not considered there. Doing so has been requested by the community before. The challenge is that incremental snapshotting needs to customize the statements, so to step through the tables in windows. Providing entire override SELECTs is at odds with that. So instead I think we'd be better off for users to just specify amendments to the WHERE clause, e.g. deleted='false'; when assembling snapshot selects, this WHERE clause amendment would be AND-ed with the other condition needed for incremental snapshotting. This furthermore has the advantage that the SELECT clause can correctly apply any given column filters, without requiring the user to accommodate for that.
Based on all that, we probly should deprecate the current property and add a new one with the semantics described above, e.g. snapshot.select.statement.amendments.