Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-549

History topic should use table metadata instead of DDLs

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Obsolete
    • Minor
    • None
    • 0.10.0.CR1
    • mysql-connector
    • None

    Description

      The approach to evolve schema using DDL commands stored in history topic is too inflexible. We should start to store table metatadata, properly keyed so compaction can be enabled.

      As a starting point we can take this proposal from rhauch

      I guess I might consider storing the table defn history (e.g., the current state/defn of each table being captured) in a new topic that uses compaction to keep only the latest version. Even if you capture 100s of tables, that would only be on the order of a hundred messages if compaction is enabled properly.

      [8:02]
      Even 1000s of tables shouldn't really be a big deal, right? Sure, it's not instantaneous, but they're all pretty small messages.

      [8:02]
      That's more scalable than storing all of them in one message (might bump up to the max message size limit on the broker), and it's far more scalable than storing all the DDL history.

      [8:03]
      Could even make it backward compatible so that if it doesn't exist at first, it is populated from the DDL history.

      Attachments

        Activity

          People

            gunnar.morling Gunnar Morling
            jpechane Jiri Pechanec
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: