Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8463

Schema information for tables during initial snapshot

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.0.5.Final
    • None
    • postgresql-connector
    • None
    • 2
    • False
    • None
    • False

      We're currently using debezium postgres connector to implement CDC for report generation information - we have multiple databases accross application, that all needs to be replicated into single separate database. We're using automatic schema creation and combination of Debezium source connector for postgres + Confluent jdbc sink connector for target db. 

      Reporting is based on SQL queries and Views, and currently we have a huge impact issue for day-one deployments - if in source database table is empty - there will be no information about this table in kafka, and because of that table will not be automatically created on target database side, making it impossible to precreate views and breaking existing SQLs for reporting. 

      I know that currently connector only focuses on providing schema changes along with data and so far the only option we found for ourselves is to create dummy records in each table and I also know that postgres doesn't support solely DDL changes via WAL. 

      But I've checked the code and I see that in fact schemas are readen for all whitelisted tables during initial snapshot, so information about the schema exists already, it's just that there is no record to send it with. 

      So could you please give me some feedback on these questions:

      • Is it possible to request feature to send schema information for empty tables during snapshot? Could it be possible that debezium sink connector supports such events?  (I assume it will be message in kafka with schema field, but payload will contain "before" and "after" as nulls)
      • Are there any other advices how to achieve automatic creation of empty tables? 

       

      Thanks in advance for answering, as we've been digging on this topic a lot lately and still couldn't find suitable solution for us. 

              Unassigned Unassigned
              numiareced Alisa Martirosyan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: