Step 1 of 4: Choose Issues


Key Summary Description

DBZ-175 changes to Snapshot and Binlog readers to allow for concurrent/partial running

Our plan for being able to add new tables to existing DBZ connectors involves effectively running a totally new snapshot and binlog reader for the tables in the new filter.
At present, the snapshot readers and binlog readers assume they will be the only snapshot/binlog process running and will be running everything that the connector is supposed to be managing (by reading from the config directly, for instance). This needs to be modified for us to be able to run these sub-connectors that manage only old filter info or new filter info.
The primary thing I imagine needs to be changed is that information about what filter info is being used should be explicitly passed in, rather than gotten from the current config. However, other things may need to be changed as well.


DBZ-175 write catch-up binlog reader

We need a "catch-up reader" for DBZ to be able to handle table additions to an existing connector's filter.
This catch up reader will merge two separate sub-connectors (two binlog reading processes within a single connector with disparate, non-intersecting filters and potentially different binlog positions information) into a single unified connector.
To do this, it will determine which sub-connector is "lagging", and read only events filtered by that sub-connector till it reaches the offset of the "leading" sub-connector.
At that point, we are free to create a new binlog reading connector with the joined filter offset information of the two sub-connectors and the binlog position that now both sub-connectors have reached. Once this connector has been created, the two sub-connectors have effectively merged into a single connector.


DBZ-175 offset information should store config filter info

offsets need to store filter info from the config to be able to evaluate whether or not the filter information in the config has changed.

blocks DBZ-175