Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-549

History topic should use table metadata instead of DDLs

    XMLWordPrintable

    Details

    • Type: Enhancement
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Out of Date
    • Affects Version/s: 0.10.0.CR1
    • Fix Version/s: None
    • Component/s: mysql-connector
    • Labels:
      None

      Description

      The approach to evolve schema using DDL commands stored in history topic is too inflexible. We should start to store table metatadata, properly keyed so compaction can be enabled.

      As a starting point we can take this proposal from Randall Hauch

      I guess I might consider storing the table defn history (e.g., the current state/defn of each table being captured) in a new topic that uses compaction to keep only the latest version. Even if you capture 100s of tables, that would only be on the order of a hundred messages if compaction is enabled properly.

      [8:02]
      Even 1000s of tables shouldn't really be a big deal, right? Sure, it's not instantaneous, but they're all pretty small messages.

      [8:02]
      That's more scalable than storing all of them in one message (might bump up to the max message size limit on the broker), and it's far more scalable than storing all the DDL history.

      [8:03]
      Could even make it backward compatible so that if it doesn't exist at first, it is populated from the DDL history.

        Attachments

          Activity

            People

            Assignee:
            gunnar.morling Gunnar Morling
            Reporter:
            jpechanec Jiri Pechanec
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: