Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3106

Forever stuck with new binlog parser (1.3 and later) when processing big JSON column data

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Critical Critical
    • 1.5.0.Beta2
    • 1.3.0.Alpha1, 1.4.1.Final
    • mysql-connector
    • None
    • False
    • False
    • Undefined
    • Hide

      create table into mysql database

      CREATE TABLE `events_blackhole` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `topic` varchar(100) NOT NULL,
  `message_key` varchar(255) DEFAULT NULL,
  `message_value` json NOT NULL,
  `message_headers` json DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;

      2. connect debezium
      3. run attached file as insert statement

      Watch debezium container CPU go 100% and no future events nor connector fail with error.

      Show
      create table into mysql database CREATE TABLE `events_blackhole` ( `id` int (11) unsigned NOT NULL AUTO_INCREMENT, `topic` varchar(100) NOT NULL, `message_key` varchar(255) DEFAULT NULL, `message_value` json NOT NULL, `message_headers` json DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8; 2. connect debezium 3. run attached file as insert statement Watch debezium container CPU go 100% and no future events nor connector fail with error.

      After upgrading debezium from 1.2 to 1.4 in production we noticed that in some cases debezium got infinitely stuck processing particular event. CPU at 100%... Restart or downgrade to 1.3 didn't solve it. After downloading 1.2 it continue as if normal.

      After discussing on gitter chat we extracted debug level logs of it and also thread dump.  From there it was clearly issue with thead that handles JSON parsing in binlog.

      So afterwards did a mysql dump and found the exact event that caused it. Also reproduce it locally.

        1. insert.sql
          4.87 MB
          Eero Koplimets

              jpechane Jiri Pechanec
              pimpelsang Eero Koplimets (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: