Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7093

Check schema length when create value to find missed DDL by SQL_BIN_LOG=OFF

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 2.5.0.Beta1
    • None
    • mysql-connector
    • None
    • False
    • None
    • False

      Feature request or enhancement

       

      Which use case/requirement will be addressed by the proposed feature?

      In mysql,

      DBA used to execute online DDL both master and replica with SQL_BIN_LOG=OFF to avoid replication lag.

      https://dev.mysql.com/doc/refman/8.0/en/innodb-online-ddl-limitations.htm

      Long running online DDL operations can cause replication lag. An online DDL operation must finish running on the source before it is run on the replica. Also, DML that was processed concurrently on the source is only processed on the replica after the DDL operation on the replica is completed.

       

      debezium couldn't notice the DDL and if the DDL is "add column",  values of the column is missed until schema recovery manually.

      we have to make debezium know there is a missed DDL.

       

      Implementation ideas (optional)

      in createValueGenerator,

      check column and row size like this if CHECK_SHCME_SIZE(input parameter) is true.

          protected StructGenerator createValueGenerator(Schema schema, TableId tableId, List<Column> columns,
                                                         ColumnNameFilter filter, ColumnMappers mappers) {   
          ...
                  return (row) -> {
                      if (CHECK_SCHEME_SIZE && columns.size() != row.length) {
                        throw new ConnectException("there is a missed DDL, do schema recovery!");
                      }
          ...

      I think CHECK_SCHEMA_SIZE is similar with database.history.skip.unparseable.ddl semantically.

      so, CHECK_SCHEMA_SIZE would be that.

       

      Limitation

      it can notice only one DDL or same operation DDLs(add or drop).

      we can't check missed DDL if execute "add column before drop column" with SQL_BIN_LOG=off, because schema size is not changed.

      and if column type or primary key is changed with SQL_BIN_LOG=off, debezium may not be able to convert values properly.

       

              Unassigned Unassigned
              hwang031451 Inki Hwang (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: