Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2922

Use collation to get charset when charset is not set

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 1.4.1.Final
    • 1.4.0.Alpha2
    • mysql-connector
    • None
    • False
    • False
    • Undefined

      After bumping debezium version to 1.4.0 Alpha2, the utf8mb4 encoding issue persists.

      Actual result in kafka is as such

      รถของคุณไม่ผ่านเà¸�ณฑ์à¸�ารสมัคร รถจัà¸�รà¸
      

      Expected result is as following.

      'รถของคุณไม่ผ่านเกณฑ์การสมัคร รถจักรยานยนต์ ต้องเป็นปี 2011 ขึ้นไป'
      

       

      Step to reproduce:

      1. Create database

      mysql> CREATE DATABASE IF NOT EXISTS dap;
      mysql> USE dap;
      mysql> show variables like "character_set_database";
      +------------------------+--------+
      | Variable_name          | Value  |
      +------------------------+--------+
      | character_set_database | latin1 |
      +------------------------+--------+
      1 row in set (0.14 sec)

      2. Create table 

      mysql> CREATE TABLE IF NOT EXISTS events ( -> id serial NOT NULL PRIMARY KEY, -> name varchar(100), -> email varchar(200) , -> message text COLLATE utf8mb4_unicode_ci, -> modified timestamp default CURRENT_TIMESTAMP NOT NULL, -> INDEX `modified_index` (`modified`) -> ); Query OK, 0 rows affected (0.04 sec)
      
      
      mysql> SHOW FULL COLUMNS FROM events; 
      


      3. Create a new connector

      curl --location --request POST 'localhost:28083/connectors' \
      --header 'Content-Type: application/json' \
      --data-raw '{
          "name": "dbz_dap_events",
          "config": {
              "connector.class": "io.debezium.connector.mysql.MySqlConnector",
              "tasks.max": "1",
              "database.hostname": "mysql",
              "database.port": "3306",
              "database.user": "confluent",
              "database.password": "confluent",
              "database.server.id": "184051",
              "database.server.name": "mysql_dap",
              "database.whitelist": "dap",
              "table.whitelist": "dap.events",
              "database.history.kafka.bootstrap.servers": "kafka:9092",
              "database.history.kafka.topic": "debz.schema-changes.mysql_dap",
              "snapshot.mode": "schema_only",
              "snapshot.locking.mode": "none"
          }
      }'

       

              Unassigned Unassigned
              victorxiang30 shuguang xiang (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: