Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2735

MySql schema change support

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Unresolved
    • Major
    • None
    • None
    • mysql-connector
    • None
    • False
    • False
    • 0
    • 0% 0%
    • Undefined

    Description

      Hello! 

      we have been using the MySQL Debezium connector in order to synchronize our production MySQL table with our Snowflake data warehouse.

      Some of our production tables store hundreds of millions of records, whenever we need to alter the schema of any of those tables we use a Percona script [pt-online-schema-change|https://www.percona.com/doc/percona-toolkit/LATEST/pt-online-schema-change.html#description. The script emulates the way that MySQL alters tables internally, but it works on a copy of the table you wish to alter. This means that the original table is not locked, and clients may continue to read and change data in it.

      From Debezium's perspective, the following process will create a queue of changes in the MySQL binlog that will cause our incoming CDC data from Debezium to be delayed by up to days or weeks. The connector is treating the alter table process as new records inserted into the table as the copy of records from the old altered table to the new schema table as new records that need to be streamed to Snowflake.

      We are looking for a way to ignore those records, without a manual intervention where we will need to go and change the connectors offset. If there was a way to signal the connector to ignore records for a certain table until we are done with the schema table changes that would be great.

       

      Let us know if you have any questions

      FYI stephen_marin gunnar.morling jpechane 

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            liorsolomon Lior Solomon (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: