Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2922

Use collation to get charset when charset is not set

    XMLWordPrintable

    Details

      Description

      After bumping debezium version to 1.4.0 Alpha2, the utf8mb4 encoding issue persists.

      Actual result in kafka is as such

      รถของคุณไม่ผ่านเà¸�ณฑ์à¸�ารสมัคร รถจัà¸�รà¸
      

      Expected result is as following.

      'รถของคุณไม่ผ่านเกณฑ์การสมัคร รถจักรยานยนต์ ต้องเป็นปี 2011 ขึ้นไป'
      

       

      Step to reproduce:

      1. Create database

      mysql> CREATE DATABASE IF NOT EXISTS dap;
      mysql> USE dap;
      mysql> show variables like "character_set_database";
      +------------------------+--------+
      | Variable_name          | Value  |
      +------------------------+--------+
      | character_set_database | latin1 |
      +------------------------+--------+
      1 row in set (0.14 sec)

      2. Create table 

      mysql> CREATE TABLE IF NOT EXISTS events ( -> id serial NOT NULL PRIMARY KEY, -> name varchar(100), -> email varchar(200) , -> message text COLLATE utf8mb4_unicode_ci, -> modified timestamp default CURRENT_TIMESTAMP NOT NULL, -> INDEX `modified_index` (`modified`) -> ); Query OK, 0 rows affected (0.04 sec)
      
      
      mysql> SHOW FULL COLUMNS FROM events; 
      


      3. Create a new connector

      curl --location --request POST 'localhost:28083/connectors' \
      --header 'Content-Type: application/json' \
      --data-raw '{
          "name": "dbz_dap_events",
          "config": {
              "connector.class": "io.debezium.connector.mysql.MySqlConnector",
              "tasks.max": "1",
              "database.hostname": "mysql",
              "database.port": "3306",
              "database.user": "confluent",
              "database.password": "confluent",
              "database.server.id": "184051",
              "database.server.name": "mysql_dap",
              "database.whitelist": "dap",
              "table.whitelist": "dap.events",
              "database.history.kafka.bootstrap.servers": "kafka:9092",
              "database.history.kafka.topic": "debz.schema-changes.mysql_dap",
              "snapshot.mode": "schema_only",
              "snapshot.locking.mode": "none"
          }
      }'

       

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            victorxiang30 shuguang xiang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: