Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2297

Catch all exception handler for event processing

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Unresolved
    • Major
    • Backlog
    • 1.2.0.Final
    • mysql-connector
    • None

    Description

      AFAIK right now there are these configuration options to change how exceptions are handled:
      *  event.processing​.failure.handling.mode
      inconsistent.schema.handling.mode

      Also could be considered related 
      database.history.skip.unparseable.ddl

      Kafka Connect side handlers:
      "errors.tolerance": "all",
      "errors.log.enable":true,
      "errors.log.include.messages":true,

      + any others?

      Yet it seems it's not enough.

      For example, when the Debezium internal database schema and MySQL don't match up, then that exception can't be handled gracefully.

      org.apache.kafka.connect.errors.ConnectException: The binlog event does not contain expected number of columns; the internal schema representation is probably out of sync with the real database schema, or the binlog contains events recorded with binlog_row_image other than FULL or the table in question is an NDB table

      To reproduce just alter some table with session sql_log_bin=0 and then insert data into that table.

      So there are two features that would help operations:

      • Catch all configurable exception handler - to tell I'm OK with data loss for only that problematic table, but please keep working on the rest of tables/databases.
      • Ability to log out what table/database actually caused that issue.

      Attachments

        Activity

          People

            Unassigned Unassigned
            pimpelsang Eero Koplimets (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: