Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-241

Topic configuration requirements are not clearly documented

    XMLWordPrintable

Details

    Description

      When you try to use Debezium's MySQL connector in a Kafka cluster configured with log compaction by default for all topics, and you let a connector's database.history.kafka.topic get auto-created, the task will die with you get confusing Kafka error messages like this one (which gets printed to Connect task status in the REST service, not to the logs!):

      org.apache.kafka.connect.errors.ConnectException: Error recording the DDL statement(s) in the database history Kakfa topic dbhistory.data-dev:0 using brokers at null: SET character_set_server=latin1, collation_server=latin1_swedish_ci;
    at io.debezium.connector.mysql.AbstractReader.wrap(AbstractReader.java:164)
    at io.debezium.connector.mysql.AbstractReader.failed(AbstractReader.java:142)
    at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:574)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.connect.errors.ConnectException: Error recording the DDL statement(s) in the database history Kakfa topic dbhistory.data-dev:0 using brokers at null: SET character_set_server=latin1, collation_server=latin1_swedish_ci;
    at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:387)
    at io.debezium.connector.mysql.SnapshotReader.execute(SnapshotReader.java:306)
    ... 1 more
Caused by: io.debezium.relational.history.DatabaseHistoryException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, or is otherwise corrupt.
    at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:175)
    at io.debezium.relational.history.AbstractDatabaseHistory.record(AbstractDatabaseHistory.java:45)
    at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:385)
    ... 2 more
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, or is otherwise corrupt.
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
    at org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
    at io.debezium.relational.history.KafkaDatabaseHistory.storeRecord(KafkaDatabaseHistory.java:165)
    ... 4 more
Caused by: org.apache.kafka.common.errors.CorruptRecordException: This message has failed its CRC checksum, exceeds the valid size, or is otherwise corrupt.

      Kafka's message is misleading; the problem is that the database history topic produces keyless messages, and log compaction requires keyed messages. Debezium's documentation should really state this, and any other requirements on the topic (e.g., is it safe to use more than one partition for this topic?)

      Attachments

        Issue Links

          relates to

          Bug - A problem that impairs or prevents the functions of the product. DBZ-239 Debezium mysql history topic with compaction enabled fails to insert messsge

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              jpechane Jiri Pechanec
              lcasillas-oportun Luis Casillas (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: