Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3168

Forever stuck with new binlog parser (1.3 and later) when processing big JSON column data

    XMLWordPrintable

    Details

    • Steps to Reproduce:
      Hide

      create table into mysql database

      CREATE TABLE `events_blackhole` (
        `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
        `topic` varchar(100) NOT NULL,
        `message_key` varchar(255) DEFAULT NULL,
        `message_value` json NOT NULL,
        `message_headers` json DEFAULT NULL,
        PRIMARY KEY (`id`)
      ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8;
      

      2. connect debezium
      3. run attached file as insert statement

      Watch debezium container CPU go 100% and no future events nor connector fail with error.

      Show
      create table into mysql database CREATE TABLE `events_blackhole` ( `id` int (11) unsigned NOT NULL AUTO_INCREMENT, `topic` varchar(100) NOT NULL, `message_key` varchar(255) DEFAULT NULL, `message_value` json NOT NULL, `message_headers` json DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=utf8; 2. connect debezium 3. run attached file as insert statement Watch debezium container CPU go 100% and no future events nor connector fail with error.

      Description

      After upgrading debezium from 1.2 to 1.4 in production we noticed that in some cases debezium got infinitely stuck processing particular event. CPU at 100%... Restart or downgrade to 1.3 didn't solve it. After downloading 1.2 it continue as if normal.

      After discussing on gitter chat we extracted debug level logs of it and also thread dump.  From there it was clearly issue with thead that handles JSON parsing in binlog.

      So afterwards did a mysql dump and found the exact event that caused it. Also reproduce it locally.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jpechanec Jiri Pechanec
              Reporter:
              pimpelsang Eero Koplimets
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: