Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3401

Implement Hybrid Mining Strategy for Oracle, seamless DDL tracking with online catalog performance

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Critical
    • 2.6.0.Beta1
    • 1.7.0.CR1
    • oracle-connector
    • None
    • False
    • False
    • Undefined

    Description

      When DML changes are written to the redo logs, these entries store table and column numeric ids that LogMiner must resolve to names. LogMiner uses the data dictionary to do this resolution; however if the data dictionary has changed and these ids no longer match, LogMiner will generate the redo SQL statements using placeholder names such as OBJ#xxxxxx for tables and COLxx for columns. We ideally should be able to resolve these redo SQL statements to correct table and column names since Oracle uses the history based schema implementation.

      The main concern with this is long running and inter-twined transactions. Lets say we have a transaction TX1 that has been running for some time and a DDL change occurs followed by some DML operations against the same newly changed table in subsequent transactions. We need to emit both the DDL schema change event as well as the changes committed in the subsequent transactions; however the offsets won't get updated until we are sure we've processed all long running transactions up to that SCN. If the connector crashes or is restarted for some reason, the connector will then reprocess all these old events once more. Since we use the history-based schema implementation, the relational table structures will be re-hydrated from the history topic. We will need a way in order to go back in time for this table such that when we reprocess all the earlier events for TX1 that we're using the correct table schema layout that was applicable at that point in time. Perhaps it's just easy enough to recover the history topic up to a given point only somehow?

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              gunnar.morling Gunnar Morling
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: