Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3760

Debezium snapshot.select.statement.overrides overrides not used

    XMLWordPrintable

Details

    • False
    • False
    • Undefined
    • Hide

      configure debezium server like below and run snapshot step

      use snapshot override for a table(or database) which has `_` in name

      config.put("%mysql.debezium.source.database.include.list", "inventory");
 config.put("debezium.source.table.include.list", "inventory.customers,inventory.orders,inventory.products,inventory.products_on_hand");
 config.put("debezium.source.snapshot.select.statement.overrides.inventory.products_on_hand", "SELECT * FROM products_on_hand WHERE 1>2");
      Show
      configure debezium server like below and run snapshot step use snapshot override for a table(or database) which has `_` in name config.put( "%mysql.debezium.source.database.include.list" , "inventory" ); config.put( "debezium.source.table.include.list" , "inventory.customers,inventory.orders,inventory.products,inventory.products_on_hand" ); config.put( "debezium.source.snapshot.select.statement.overrides.inventory.products_on_hand" , "SELECT * FROM products_on_hand WHERE 1>2" );

    Description

      seems like with debezium 1.6.Final  `snapshot.select.statement.overrides` is not picked up correctly

      `_`. character in the table names replaced by `.` dot which might be source of  the issue

      adding below Mysql unit-test log which shows the issue

       

       2021-07-15 09:23:25,213 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) database.server.name = testc
>>>> 2021-07-15 09:23:25,213 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) snapshot.select.statement.overrides.inventory.products.on.hand = SELECT * FROM products_on_hand WHERE 1>2
 2021-07-15 09:23:25,214 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) transforms.unwrap.add.fields = op,table,source.ts_ms,db
 2021-07-15 09:23:25,214 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) table.include.list = inventory.customers,inventory.orders,inventory.products,inventory.test_date_table,inventory.products_on_hand,inventory.geom,inventory.table_datatypes
 2021-07-15 09:23:25,214 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) database.history = io.debezium.relational.history.FileDatabaseHistory
 2021-07-15 09:23:25,215 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) database.include.list = inventory

..........

2021-07-15 09:23:26,772 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Reading structure of database 'inventory'
 2021-07-15 09:23:26,901 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Snapshot step 6 - Persisting schema history
 2021-07-15 09:23:26,970 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Releasing global read lock to enable MySQL writes
 2021-07-15 09:23:26,982 INFO [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Writes to MySQL tables prevented for a total of 00:00:01.132
 2021-07-15 09:23:26,982 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Snapshot step 7 - Snapshotting data
 2021-07-15 09:23:26,983 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Snapshotting contents of 5 tables while still in transaction
........

2021-07-15 09:23:27,073 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Exporting data from table 'inventory.products_on_hand' (4 of 5 tables)

>>>> 2021-07-15 09:23:27,073 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) For table 'inventory.products_on_hand' using select statement: 'SELECT * FROM `inventory`.`products_on_hand`'
 2021-07-15 09:23:27,093 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Finished exporting 9 records for table 'inventory.products_on_hand'; total duration '00:00:00.021'

       

       

      this lines shows the issue

      >> 2021-07-15 09:23:25,213 INFO [io.deb.con.com.BaseSourceTask] (pool-11-thread-1) snapshot.select.statement.overrides.inventory.products.on.hand = SELECT * FROM products_on_hand WHERE 1>2 

      select is not overriden

      >> 2021-07-15 09:23:27,073 INFO [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) For table 'inventory.products_on_hand' using select statement: 'SELECT * FROM `inventory`.`products_on_hand`'

       cc: gunnar.morling

      Attachments

        Issue Links

          is caused by

          Bug - A problem that impairs or prevents the functions of the product. DBZ-2622 Not able to configure Debezium Server via smallrye/microprofile environment variables

          • Minor - To be worked after urgent, high, and medium priorities are resolved. This term is chosen when the severity of the issue is low, or the complexity or effort to correct it may be higher, relatively speaking. For low priority issues, known workarounds exist or are not needed due to the trivial effort needed to address the issue.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. DBZ-3817 Debezium Server incorrectly casts MySQL connection property to lowercase

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              anmohant Anisha Mohanty
              ismailsimsek ismail simsek
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: