Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-29

Handle large messages

XMLWordPrintable

      Most Kafka clusters run with a max message size of 1MB - 2MB. I'm not sure if/how Debezium/Kafka Connect handle this now, but there should be a relief valve for messages that are larger than the limit. Common pitfalls to this are tables that have large TEXT or BLOB columns.

      Some ideas:

      1. Add the ability to project out certain columns on a per-table basis. Users could then filter out the offending columns.
      2. Add the ability to drop the messages if they're too large.
      3. Add the ability to truncate text/blob columns.

      I'm partial to have options both (1) and (3). (2) is not something that I would use.

              rhauch Randall Hauch (Inactive)
              criccomini Chris Riccomini (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: