Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-4628

Support incremental materialization refresh strategies

    XMLWordPrintable

Details

    • Feature Request
    • Resolution: Done
    • Major
    • None
    • None
    • Query Engine, Server
    • None

    Description

      Further enhance/refine the built in materialization management to support alternative invalidation and refresh strategies.

      In terms of priority, this would target external materialization first.

      There must be an alternative to the implied default of the table rename strategy - which is problematic when a source is not transactional and should be changed to a merge/delete approach.

      The general approaches are lazy (rows marked as dirty with refresh on query/schedule) and eager (which can even attempt to maintain close to transactional consistency). Optionally incremental loading can be used as well, to only load the portions of the materialization that are requested.

      This requires more logic that we currently have:

      • mapping source level row events to row level effects in materialized views (at worst we can refresh the whole row)
      • additional status information possibly down to a column level, which much be added to the materialization table.
      • incremental loading requires tracking the effectively covered predicate ranges and even columns of what has been loaded

      It would probably be good to focus on a single scenario at first, such as:

      external materialization, with non-incremental load (to reuse the existing simplistic load, but not rename logic), and eager refresh - and only considering transitive row effects based upon key relationships.

      Then we can ensure that sufficient options are being added to account for additional scenarios.

      Attachments

        Issue Links

          Activity

            People

              rhn-engineering-rareddy Ramesh Reddy
              rhn-engineering-shawkins Steven Hawkins
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: