Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1086

Debezium fails when consuming table without primary key with turned on topic routing

    XMLWordPrintable

    Details

      Description

      I have set up debezium with topic routing enabled.
      After adding a new table to connector configuration I've received the following error:

      [2019-01-16 19:15:20,776] ERROR WorkerSourceTask{id=app1-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:177)
      org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
      	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
      	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
      	at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:44)
      	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:292)
      	at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)
      	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
      	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.kafka.connect.errors.DataException: Only Struct objects supported for [Updating schema], found: null
      	at org.apache.kafka.connect.transforms.util.Requirements.requireStruct(Requirements.java:52)
      	at io.debezium.transforms.ByLogicalTableRouter.apply(ByLogicalTableRouter.java:184)
      	at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:44)
      	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
      	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
      	... 11 more
      

      The reason for the issue is that the new table doesn't have a primary key and topic routing is enabled. So, debezium tries to add an original topic name to message key, but message key is `null` (because the table has no primary key).

      Is there any way to use topic routing with a table without a primary key?

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            ikovbas Ivan Kovbas (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: