Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-928

Discard null fields in MongoDB event flattening SMT

    XMLWordPrintable

Details

    Description

      When trying unwrap json document from mongo and documet having a field name hour, the unwrapping doesnt work with below error

      org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
              at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
              at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
              at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:44)
              at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:292)
              at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:228)
              at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
              at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.kafka.connect.errors.DataException: hour is not a valid field name
              at org.apache.kafka.connect.data.Struct.lookupField(Struct.java:254)
              at org.apache.kafka.connect.data.Struct.put(Struct.java:202)
              at io.debezium.connector.mongodb.transforms.MongoDataConverter.convertFieldValue(MongoDataConverter.java:220)
              at io.debezium.connector.mongodb.transforms.MongoDataConverter.convertFieldValue(MongoDataConverter.java:130)
              at io.debezium.connector.mongodb.transforms.MongoDataConverter.convertRecord(MongoDataConverter.java:45)
              at io.debezium.connector.mongodb.transforms.UnwrapFromMongoDbEnvelope.apply(UnwrapFromMongoDbEnvelope.java:199)
              at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:44)
              at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
              at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
      

      config file:

      name=mongoom
      connector.class=io.debezium.connector.mongodb.MongoDbConnector
      tasks.max=1
      database.list=xxxxxx
      connection.url=xxxxx
      mongodb.hosts=xxxxx
      mongodb.name=ordermanagement
      collection.whitelist=redmart.order
      database.whitelist=redmart
      converter.class=org.apache.kafka.connect.json.JsonConverter
      key.converter=org.apache.kafka.connect.json.JsonConverter
      value.converter=org.apache.kafka.connect.json.JsonConverter
      key.converter.schemas.enable=false
      value.converter.schemas.enable=false
      internal.value.converter=org.apache.kafka.connect.json.JsonConverter
      #plugin.path=share/java/
      transforms=unwrap
      transforms.unwrap.type=io.debezium.connector.mongodb.transforms.UnwrapFromMongoDbEnvelope
      transforms.unwrap.array.encoding=document
      transforms.unwrap.flatten.struct=true
      transforms.unwrap.flatten.struct.delimiter=_
      

      The issue seems to be occuring when the value in the field is of Null type.

      I have gone through the code and seems like it says it does not support.

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            hariprasad1 Hariprasad A (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: