Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4032

Crash processing MariaDB DATETIME fields returns empty blob instead of null (Snapshotting with useCursorFetch option)

    XMLWordPrintable

Details

    Description

      Linked to DBZ-3773 as per crancran on Discussion 

       

      While trying to run Debezium MySQL connector on a MariaDB 10.0. When using "snapshot.fetch.size" = "1000", the connector fails with the following error:

      Stack Trace:

       

      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      	at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:42)
      	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:130)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: io.debezium.DebeziumException: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0
      	at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:78)
      	at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:113)
      	... 5 more
      Caused by: java.lang.RuntimeException: Invalid length when read MySQL DATETIME value. BIN_LEN_DATETIME is 0
      	at io.debezium.connector.mysql.MysqlBinaryProtocolFieldReader.readTimestampField(MysqlBinaryProtocolFieldReader.java:86)
      	at io.debezium.connector.mysql.AbstractMysqlFieldReader.readField(AbstractMysqlFieldReader.java:39)
      	at io.debezium.connector.mysql.MySqlConnection.getColumnValue(MySqlConnection.java:571)
      	at io.debezium.jdbc.JdbcConnection.rowToArray(JdbcConnection.java:1470)
      	at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:355)
      	at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:306)
      	at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:136)
      	at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:69)
      	... 6 more
      

      My understanding is that this setting will automatically enable useCursorFetch = true (changing the connection protocol from TextProtocol to BinaryProtocl).

       

      Table DDL:

      CREATE TABLE `collaborator` (
        `id` int(11) NOT NULL AUTO_INCREMENT,
        `order_id` int(11) DEFAULT NULL,
        `agency_id` int(11) DEFAULT NULL,
        `parent_id` int(11) DEFAULT NULL,
        `actor_profile_id` int(11) DEFAULT NULL,
        `order_external_reference` varchar(255) COLLATE utf8_unicode_ci DEFAULT NULL,
        `created_at` datetime NOT NULL COMMENT '(DC2Type:datetime_immutable)',
        PRIMARY KEY (`id`),
        KEY `IDX_606D487C8D9F6D38` (`order_id`),
        KEY `IDX_606D487CCDEADB2A` (`agency_id`),
        KEY `IDX_606D487C727ACA70` (`parent_id`),
        KEY `IDX_606D487CFED607AB` (`actor_profile_id`),
        KEY `order_external_reference_idx` (`order_external_reference`),
        CONSTRAINT `FK_606D487C727ACA70` FOREIGN KEY (`parent_id`) REFERENCES `order_actors` (`id`),
        CONSTRAINT `FK_606D487C8D9F6D38` FOREIGN KEY (`order_id`) REFERENCES `orders` (`id`),
        CONSTRAINT `FK_606D487CFED607AB` FOREIGN KEY (`actor_profile_id`) REFERENCES `stakeholder_profile` (`id`)
      ) ENGINE=InnoDB AUTO_INCREMENT=4338568 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
      

       

      Related connector config:

       

      "event.deserialization.failure.handling.mode" = "warn"
      "include.schema.changes" = "true"
      "read.only" = "true"
      "snapshot.locking.mode" = "none"
      "snapshot.mode" = "when_needed"
      "snapshot.fetch.size" = "1000"

       

      Thank you!

       

      Attachments

        Issue Links

          Activity

            People

              ccranfor@redhat.com Chris Cranford
              omar.ghalawinji@gmail.com Omar Ghalawinji
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: