Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6712

schema.history.internal.store.only.captured.databases.ddl flag not considered while snapshot schema to history topic

XMLWordPrintable

    • False
    • None
    • False
    • Moderate

      Bug report

      What Debezium connector do you use and what version?

      2.2.1Final (same code exist also at latest)

      What is the connector configuration?

      relevant configuration:

      database.include.list: <database_name>
      connector.class: io.debezium.connector.mysql.MySqlConnector
      schema.history.internal.store.only.captured.databases.ddl: true
      schema.history.internal.store.only.captured.tables.ddl: false
      snapshot.include.collection.list: <database_name>.*
      snapshot.mode: schema_only\schema_only_recovery
      table.include.list: <database_name>.table_test1,<database_name>.table_test2,<database_name>.cdc_dummy

       

      What is the captured database version and mode of depoyment?

      MySql v5.7

      What behaviour do you expect?

      I accept the schema.history.internal.store.only.captured.databases.ddl To be respected while snapshot is filtering out tables and collect only included databases tables structure into the <connector_name>.history topic.

      What behaviour do you see?

      debezium is collecting all databases tables into history topic

      Do you see the same behaviour using the latest relesead Debezium version?

      relevant Code as not been changed to support it  (RelationalTableFilters.java,RelationalSnapshotChangeEventSource.java)

      was recreated also in 2.3.0Final version.

       

      How to reproduce the issue using our tutorial deployment?

      running the above configuration on MySQL Instance which contains more than one application database. but include only one of the databases in the database.include.list

      1. Reset and run a connector to take a new snapshot.
      2. Add a column to one of the tables in the second (untracked) database.
      3. make som DML changes to the tracked database make sure new binlog offset is recorded in the connector offset topic.
      4. restart the connector
      5. Alter column type - parseException should occur.

      Feature request or enhancement

      schema.history.internal.store.only.captured.databases.ddl  flag to control the include of only tracked databases(include.database.list) ddl in history topic.

      Which use case/requirement will be addressed by the proposed feature?

      There are several methods this situation can be prevented but consider the documentation regarding schema.history.internal.store.only.captured.databases.ddl   flag it should be handle on that level.

      Without this behavior for stability you must include schema.history.internal.store.only.captured.tables.ddl and perform a schema_only_recovery procedure whenever new table is added to the include list.

      Implementation ideas (optional)

      the quickest and cleanest way is to fix it in RelationalTableFilters.java

      the eligibleForSchemaDataCollectionFilter should return a schemaSnapshotFilter which takes into consideration the  schema.history.internal.store.only.captured.databases.ddl flag value

            Unassigned Unassigned
            matanc Matan Cohen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: