Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3645

Change CLOB/BLOB data type support to an opt-in feature

    XMLWordPrintable

Details

    • Task
    • Resolution: Done
    • Blocker
    • 1.6.0.CR1
    • 1.6.0.Beta2
    • oracle-connector
    • None

    Description

      When supporting CLOB / BLOB data types, the connector performs several expensive tasks such as needing to reconcile and merge events in the transaction buffer at commit time, must read large volumes of data from the LogMiner view, and needs to manage the SCN range that is provided to LogMiner differently than what we did before introducing CLOB / BLOB support.

      As a result, we've decided that this feature should be an opt-in only option for now. In addition to making it opt-in, there was a suggestion to try a hybrid approach with the SCN ranges used while mining such that we would use the sliding window algorithm that existed prior to adding CLOB / BLOB support and if we detect a transaction that commits that has a starting SCN that comes before the current window's starting point, we would rerun an additional mining query in order to guarantee we got the full picture of that transaction to satisfy the CLOB / BLOB requirements. Using this hybrid approach, we avoid the excessive re-mining tax of starting from the oldest active transaction on each iteration.

      See attached image of what the mining process looks like currently with CLOB / BLOB and why it needs to be optimized.

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              ccranfor@redhat.com Chris Cranford
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: