Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-983

BigDecimal has mismatching scale value for given Decimal schema error due to permissive mysql ddl

    Details

      Description

      Seems we have run into an issue when we have changed the default value of our source mysql table column to a value not valid for avro it seems. The exact ddl statement issued is recorded in the dbhistory topic:

      dbhistory topic content:

      {
        "source" : {
          "server" : "test"
        },
        "position" : {
          "ts_sec" : 1542210683,
          "file" : "mysql-bin.000289",
          "pos" : 1017928405,
          "server_id" : 3401,
          "event" : 1
        },
        "databaseName" : "test",
        "ddl" : "alter table joel2 modify d decimal(10,5) default 0.0 not null"
      }
      

      However, the ddl should be the same as what the mysql server implicitly converts:

      mysql- [test]> show create table joel2;
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | Table | Create Table                                                                                                                                                                                                                                        |
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      | joel2 | CREATE TABLE `joel2` (
        `a` int(11) NOT NULL,
        `b` text,
        `c` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
        `d` decimal(10,5) NOT NULL DEFAULT '0.00000',
        PRIMARY KEY (`a`)
      ) ENGINE=InnoDB DEFAULT CHARSET=utf8 |
      +-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
      1 row in set (0.00 sec)
      
      

      Error (from schema registry?) when debezium tries to evolve the schema:

      {
      "name": "testSchemaChanges-source3",
      "connector": {
      "state": "RUNNING",
      "worker_id": "172.16.230.69:8083"
      },
      "tasks": [
      {
      "state": "FAILED",
      "trace": "org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:269)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:293)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:745)\nCaused by: org.apache.kafka.connect.errors.DataException: BigDecimal has mismatching scale value for given Decimal schema\n\tat org.apache.kafka.connect.data.Decimal.fromLogical(Decimal.java:68)\n\tat io.confluent.connect.avro.AvroData$5.convert(AvroData.java:261)\n\tat io.confluent.connect.avro.AvroData.defaultValueFromConnect(AvroData.java:951)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:856)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:925)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:832)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:820)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.addAvroRecordField(AvroData.java:925)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:832)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:685)\n\tat io.confluent.connect.avro.AvroData.fromConnectSchema(AvroData.java:680)\n\tat io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:325)\n\tat io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:75)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:269)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)\n\tat org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)\n\t... 11 more\n",
      "id": 0,
      "worker_id": "172.16.230.67:8083"
      }
      ],
      "type": "source"
      }
      

      Very easy to reproduce:

      1. create table t1 (a int, b decimal);
      2. insert into t1 values (1,1.1);
      3. start debezium connector
      4. alter table t1 modify b decimal(10,5) default 0.0 not null;
      5. insert into t1 (a) values (2);
      6. connector should now be in a failed state

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                joel.salmeron Joel Salmeron Viver
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: