Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-142

NVARCHAR columns are not supported

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 0.3.6, 0.4
    • 0.3.4
    • mysql-connector
    • None
    • Hide

      1. Run debezium configured for a MySQL database.
      2. Create a table with a NVARCHAR column:

      CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);

      3. Check the debezium log output

      Show
      1. Run debezium configured for a MySQL database. 2. Create a table with a NVARCHAR column: CREATE TABLE test2 (id INT PRIMARY KEY , someColumn NVARCHAR(100) NOT NULL ); 3. Check the debezium log output

      Debezium seems to have problems when columns with type NVARCHAR appears in the binlog. NVARCHAR is equivalent to a VARCHAR CHARACTER SET utf8 (http://dev.mysql.com/doc/refman/5.7/en/charset-national.html).

      It works fine when the table exists when starting Debezium, then for the initial snapshot it is parsed as a regular VARCHAR and everything is OK:

      CREATE TABLE test (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);

      2016-10-26 08:36:08,757] INFO 	CREATE DATABASE test (io.debezium.connector.mysql.SnapshotReader:569)
[2016-10-26 08:36:08,799] INFO 	USE test (io.debezium.connector.mysql.SnapshotReader:569)
[2016-10-26 08:36:08,836] INFO 	CREATE TABLE `test` (
  `id` int(11) NOT NULL,
  `someColumn` varchar(100) CHARACTER SET utf8 NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 (io.debezium.connector.mysql.SnapshotReader:569)
[2016-10-26 08:36:08,876] INFO Step 7: releasing global read lock to enable MySQL writes (io.debezium.connector.mysql.SnapshotReader:325)

      But, if I create a new table with Debezium running with a NVARCHAR, so that we get the CREATE TABLE command in the binlog stream, then I get this error:

      CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL);

      [2016-10-26 08:36:50,373] ERROR Error parsing DDL statement and updating tables: CREATE TABLE test2 (id INT PRIMARY KEY, someColumn NVARCHAR(100) NOT NULL) (io.debezium.connector.mysql.MySqlSchema:328)
io.debezium.text.ParsingException: Expecting ')' at line 1, column 60 but found '(':  someColumn NVARCHAR ===>> (100) NOT NULL)
	at io.debezium.text.TokenStream.consume(TokenStream.java:694)
	at io.debezium.connector.mysql.MySqlDdlParser.parseCreateDefinitionList(MySqlDdlParser.java:548)
	at io.debezium.connector.mysql.MySqlDdlParser.parseCreateTable(MySqlDdlParser.java:365)
	at io.debezium.connector.mysql.MySqlDdlParser.parseCreate(MySqlDdlParser.java:271)
	at io.debezium.connector.mysql.MySqlDdlParser.parseNextStatement(MySqlDdlParser.java:146)
	at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:285)
	at io.debezium.relational.ddl.DdlParser.parse(DdlParser.java:266)
	at io.debezium.connector.mysql.MySqlSchema.applyDdl(MySqlSchema.java:326)
	at io.debezium.connector.mysql.BinlogReader.handleQueryEvent(BinlogReader.java:390)
	at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:295)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:877)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:735)
	at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:451)
	at com.github.shyiko.mysql.binlog.BinaryLogClient$5.run(BinaryLogClient.java:632)
	at java.lang.Thread.run(Thread.java:745)

      I assume this applies to the other synonyms for NVARCHAR listed here as well: http://dev.mysql.com/doc/refman/5.7/en/charset-national.html

              depe_jira Dennis Persson (Inactive)
              depe_jira Dennis Persson (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: