Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5529

Problem in ExtractNewRecordState in Sink using debezium struct

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      Bug report

      When the sink connector consumes events from topic POC.PALADIN.TABLE_5 using io.debezium.transforms.ExtractNewRecordState, it is receiving the following error:

       

      ERROR  ||  WorkerSinkTask{id=sink-postgres-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: io.debezium.data.VariableScaleDecimal (STRUCT) type doesn't have a mapping to the SQL database column type   [org.apache.kafka.connect.runtime.WorkerSinkTask]
      org.apache.kafka.connect.errors.ConnectException: io.debezium.data.VariableScaleDecimal (STRUCT) type doesn't have a mapping to the SQL database column type

      The table structure is as follows:

      CREATE TABLE PALADIN.TABLE_5(
           ID NUMBER NOT NULL,
           VARCHAR1 VARCHAR2(500) NOT NULL,
           NUMBER1 NUMBER(8,2) NOT NULL,
           CLOB1 CLOB,
           DATE1 DATE,
           PRIMARY KEY (id)
      );

      After some attempts to solve the problem I found that changing my table structure to the following:

      CREATE TABLE PALADIN.TABLE_5(
           ID NUMBER(8,2) NOT NULL,
           VARCHAR1 VARCHAR2(500) NOT NULL,
           NUMBER1 NUMBER(8,2) NOT NULL,
           CLOB1 CLOB,
           DATE1 DATE,
           PRIMARY KEY (id)
      );
      By putting length and precision in the ID field, the sink connector worked normally.
       
      Why is this transform behaving this way?

      What Debezium connector do you use and what version?

      2.0

      What is the connector configuration?

      {
         "name":"oracle-debezium",
         "config":

      {       "connector.class":"io.debezium.connector.oracle.OracleConnector",       "database.hostname":"x.x.x.x",       "database.port":"1521",       "database.user":"c##miner",       "database.password":"pass",       "database.dbname":"POC",       "database.pdb.name":"POC1",       "database.server.name":"POC",       "log.mining.strategy":"online_catalog",       "database.history.kafka.bootstrap.servers":"broker:29092",       "database.history.kafka.topic":"schema-changes.inventory",       "tombstones.on.delete":"true",       "lob.enabled":"TRUE",       "table.include.list":"PALADIN.TABLE_5"    }

      }

      What is the connector sink configuration?

      {
          "name": "sink-postgres",
          "config":

      {         "name": "sink-postgres",         "connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",         "tasks.max": 1,         "connection.url": "jdbc:postgresql://x.x.x.x:7341/Mars",         "connection.user": "cool",         "connection.password": "pass",         "topics": "POC.PALADIN.TABLE_5",         "table.name.format": "TABLE_5",         "insert.mode": "upsert",         "pk.mode": "record_key",         "pk.fields": "ID",         "delete.enabled": "true",         "transforms": "unwrap",         "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",         "transforms.unwrap.drop.tombstones": "false",         "auto.create": "true"     }

      }

      What is the captured database version and mode of depoyment?

      Source: Oracle 19c in contêiner mode

      Sink: Postgres 14 in conteiner mode

      What behaviour do you expect?

      Capture events in oracle database and write this events in topic  named POC.PALADIN.TABLE_5. After this the sink read the topic and inject events in postgres database.

      What behaviour do you see?

      When the sink read the events in topic is getting this exception: Error: io.debezium.data.VariableScaleDecimal (STRUCT) type doesn't have a mapping to the SQL database column type

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      Attachments

        1. event getting a error.json
          5 kB
          Alexandre Costa
        2. event with right behavior.json
          5 kB
          Alexandre Costa
        3. sink-error.log
          7 kB
          Alexandre Costa

        Activity

          People

            Unassigned Unassigned
            alexandre.neo Alexandre Costa (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: