We have some MySQL DBs that have many tables. These DBs sometimes have new tables added to them that we'd like to load into Kafka with Debezium. We also use whitelists in Debezium to list only the tables that we want to load. If the table is created before it's added to the DBZ whitelist, we hit an issue where we need to snapshot the new table in Debezium. Currently, this is not possible unless we rebootstrap the entire DB with Debezium from scratch. This can lead to very long delays as the whole DB is rebootstrapped, and also causes a ton of duplicate data that needn't been loaded in.
There are several work arounds to this:
- Require that all new tables get whitelisted in DBZ before they're added to the actual DB.
- Don't use white lists.
- Create a separate DBZ connector instance that runs just against the new tables.
For (3), you can eventually consolidate the new DBZ connector with the old one by pausing them both at the exact same spot in the binlog (presumably by pausing the slave that they're reading off of, and watiting).