-
Feature Request
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
None
-
False
Feature request or enhancement
Which use case/requirement will be addressed by the proposed feature?
In mysql,
DBA used to execute online DDL both master and replica with SQL_BIN_LOG=OFF to avoid replication lag.
https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-limitations.htm
Long running online DDL operations can cause replication lag. An online DDL operation must finish running on the source before it is run on the replica. Also, DML that was processed concurrently on the source is only processed on the replica after the DDL operation on the replica is completed.
debezium couldn't notice the DDL and if the DDL is "add column", values of the column is missed until schema recovery manually.
we have to make debezium know there is a missed DDL.
Implementation ideas (optional)
check column and row size like this if CHECK_SHCME_SIZE(input parameter) is true.
protected StructGenerator createValueGenerator(Schema schema, TableId tableId, List<Column> columns, ColumnNameFilter filter, ColumnMappers mappers) { ... return (row) -> { if (CHECK_SCHEME_SIZE && columns.size() != row.length) { throw new ConnectException("there is a missed DDL, do schema recovery!"); } ...
I think CHECK_SCHEMA_SIZE is similar with database.history.skip.unparseable.ddl semantically.
so, CHECK_SCHEMA_SIZE would be that.
Limitation
it can notice only one DDL or same operation DDLs(add or drop).
we can't check missed DDL if execute "add column before drop column" with SQL_BIN_LOG=off, because schema size is not changed.
and if column type or primary key is changed with SQL_BIN_LOG=off, debezium may not be able to convert values properly.
- relates to
-
DBZ-7143 Error when fail converting value with internal schema
- Resolved
- links to
-
RHEA-2024:129636 Red Hat build of Debezium 2.5.4 release