Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7600

Allow schema_only_recovery where history exists

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Obsolete
    • Icon: Minor Minor
    • 2.6.0.Beta1
    • None
    • oracle-connector
    • None
    • False
    • None
    • False

      Hi there,

      This is related to the conversation had here: https://debezium.zulipchat.com/#narrow/stream/348250-community-oracle/topic/Allow.20schema_only_recovery.20where.20history.20exists

      Currently, schema_only_recovery mode only works when the schema history does not exist. The below is for Oracle (I'm not certain if it's the same for the other connectors), where the test for the schema existing happens before the test on the snapshot mode.

             if (!schema.historyExists()) {
                  LOGGER.warn("Database schema history was not found but was expected");
                  if (config.getSnapshotMode().shouldSnapshotOnSchemaError()) {
                      LOGGER.info("The db-history topic is missing but we are in {} snapshot mode. " +
                              "Attempting to snapshot the current schema and then begin reading the redo log from the last recorded offset.",
                              OracleConnectorConfig.SnapshotMode.SCHEMA_ONLY_RECOVERY);
                  }
      

      In some cases, a connector is placed into schema_only_recovery mode for reasons other than loss or corruption of the history topic/file. For example, when a new table is added for which we wish to perform an incremental snapshot (which requires the schema to be generated for the new table).

      In our environment, we use CI/CD to do updates to the application.properties file and devs don't have direct access to the production server. Because of this, we have a script that checks for newly added tables and then removes the history file on disk and then puts the connector into schema_only_recovery to regenerate all the schemas.

      Ideally, we would like for Debezium to handle this itself rather than us hacking something together as a workaround i.e. the option to force Debezium to remove the schema history and then perform the schema_only_recovery.

              Unassigned Unassigned
              nathan-smit-1 Nathan Smit
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: