Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4213

ConvertingEngineBuilder looses the accents

    XMLWordPrintable

Details

    • False
    • False
    • Hide

      I tested with  debezium server with postgreSQL & wal2json.

      You can reproduce it If you add a new row in the database with one string column with  accents in the field.

      You should see the bytes. keep accents up to this point and then lose accents in the last step.

      "value != null ? new String(value)" line 174 in ConvertingEngineBuilder

      Show
      I tested with  debezium server with postgreSQL & wal2json. You can reproduce it If you add a new row in the database with one string column with  accents in the field. You should see the bytes. keep accents up to this point and then lose accents in the last step. "value != null ? new String(value)" line 174 in ConvertingEngineBuilder

    Description

      Affect from 1.6 to 1.8

      public class ConvertingEngineBuilder<R> implements Builder<R> {
      
      @SuppressWarnings("unchecked")
      @Override
      public DebeziumEngine<R> build() {
      .
      .
      .
      keyConverter = createConverter(formatKey, true);
      valueConverter = createConverter(formatValue, false);
      
      
      toFormat = (record) -> {
          final byte[] key = keyConverter.fromConnectData(TOPIC_NAME, record.keySchema(), record.key());
      
      final byte[] value = valueConverter.fromConnectData(TOPIC_NAME, record.valueSchema(), record.value());
      // Here the 'value' and record.value() has accents
      
          return isFormat(formatKey, Json.class) && isFormat(formatValue, Json.class)
                  || isFormat(formatValue, CloudEvents.class)
      // Here, with "new String(value)" you are loosing accents
      // You can check the bytes between "byte[] value" & "new String(value) bytes"
      // You can replace by "new String(value, StandardCharsets.UTF_8)" to fix it
                          ? (R) new EmbeddedEngineChangeEvent<String, String>(
                                  key != null ? new String(key) : null,
                                  value != null ? new String(value) : null,
                                  record)
                          : (R) new EmbeddedEngineChangeEvent<byte[], byte[]>(
                                  key,
                                  value,
                                  record);
      };
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lordbios83 Jaume Grané Sánchez (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: