Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2668

BigDecimal has mismatching scale value for given Decimal schema error due to permissive sqlserver ddl

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Rejected
    • 0.8.3.Final
    • None
    • sqlserver-connector
    • None

    Description

      Seems we have run into an issue when we have changed the default value of our source mysql table column to a value not valid for avro it seems. The exact ddl statement issued is recorded in the dbhistory topic:

      dbhistory topic content:

      {
        "source" : {
          "server" : "test"
        },
        "position" : {
          "ts_sec" : 1542210683,
          "file" : "mysql-bin.000289",
          "pos" : 1017928405,
          "server_id" : 3401,
          "event" : 1
        },
        "databaseName" : "test",
        "ddl" : "alter table joel2 modify d decimal(10,5) default 0.0 not null"
      }
      

      However, the ddl should be the same as what the mysql server implicitly converts:

      sql- [test]> show create table joel2;
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | Table | Create Table                                                                                                                                                                                                                                        |
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | joel2 | CREATE TABLE `joel2` (
        `a` int(11) NOT NULL,
        `b` text,
        `c` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
        `d` decimal(10,5) NOT NULL DEFAULT '0.00000',
        PRIMARY KEY (`a`)
      ) 
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      1 row in set (0.00 sec)
      
      

      Error (from schema registry?) when debezium tries to evolve the schema:

      {
      "name": "testSchemaChanges-source3",
      "connector": {
      "state": "RUNNING",
      "worker_id": "172.16.230.69:8083"
      },
      "tasks": [
      {
      "state": "FAILED",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:269)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.kafka.connect.errors.DataException: BigDecimal has mismatching scale value for given Decimal schema\n\tat org.apache.kafka.connect.data.Decimal.fromLogical(Decimal.java:68)\n\tat io.confluent.connect.avro.AvroData$5.convert(AvroData.java:261)\n\tat io.confluent.connect.avro.AvroData.defaultValueFromConnect(AvroData.java:951)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:856)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:925)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:832)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:820)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:925)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:832)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:680)\n\tat io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:325)\n\tat io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:75)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:269)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)\n\t... 11 more\n",
      "id": 0,
      "worker_id": "172.16.230.67:8083"
      }
      ],
      "type": "source"
      }
      

      Very easy to reproduce:

      1. create table t1 (a int, b decimal);
      2. insert into t1 values (1,1.1);
      3. start debezium connector
      4. alter table t1 modify b decimal(10,5) default 0.0 not null;
      5. insert into t1 (a) values (2);
      6. connector should now be in a failed state

      Attachments

        Issue Links

          Activity

            People

              jpechane Jiri Pechanec
              idrees.mohammed Idrees Mohammed (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: