-
Feature Request
-
Resolution: Unresolved
-
Major
-
None
-
None
-
False
-
None
-
False
Add the option to limit changes read by number of changes rather than number of transactions.
Currently, the SQL Server connector can limit changes read by transaction with the max.iteration.transactions option. This request is to add the ability to limit by changes to avoid unpredictable memory usage that occurs when a single transaction (or small number of them) may have many changes.
One possibility for implementation would be to introduce an integer `max.changes.read` option. The option would be used to limit the number of records returned by the queries run in the getChangesForTables function, and would need to loop until all changes have been retrieved for the LSN range for the table.
The `executeIteration` function would also need to be modified. It would need to disptach data change events for each batch of changes to avoid the memory consumption associated with ingesting all changes at once. The process of retrieving and dispatching changes would likely need to be completed for the LSN range for each table, which would mean data change events would be ordered at a table level and not ordered across tables.