The approach to evolve schema using DDL commands stored in history topic is too inflexible. We should start to store table metatadata, properly keyed so compaction can be enabled.

As a starting point we can take this proposal from rhauch

I guess I might consider storing the table defn history (e.g., the current state/defn of each table being captured) in a new topic that uses compaction to keep only the latest version. Even if you capture 100s of tables, that would only be on the order of a hundred messages if compaction is enabled properly.

[8:02]
Even 1000s of tables shouldn't really be a big deal, right? Sure, it's not instantaneous, but they're all pretty small messages.

[8:02]
That's more scalable than storing all of them in one message (might bump up to the max message size limit on the broker), and it's far more scalable than storing all the DDL history.

[8:03]
Could even make it backward compatible so that if it doesn't exist at first, it is populated from the DDL history.