Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4704

Cannot expand JSON payload with nested arrays of objects

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.9.0.Alpha2
    • 1.8.0.Final
    • mysql-connector
    • None
    • False
    • False
    • Hide

      I won't provide instructions on how to reproduce the issue on a "live" environment but instead here are the instructions on how to replicate it in the `EventRouterTest` test class in the Debezium project:
      Change the `payload` for the `eventRecord` local variable in the `canExpandJsonPayloadIfConfigured` to the following one which contains an array of objects instead of an array of strings:

      "{\"fullName\": \"John Doe\", \"enabled\": true, \"rating\": 4.9, \"age\": 42, \"pets\": [{\"type\": \"dog\"}, {\"type\": \"cat\"}]}"

      and run the test

      Show
      I won't provide instructions on how to reproduce the issue on a "live" environment but instead here are the instructions on how to replicate it in the `EventRouterTest` test class in the Debezium project: Change the `payload` for the `eventRecord` local variable in the `canExpandJsonPayloadIfConfigured` to the following one which contains an array of objects instead of an array of strings: "{\" fullName\ ": \" John Doe\ ", \" enabled\ ": true , \" rating\ ": 4.9, \" age\ ": 42, \" pets\ ": [{\" type\ ": \" dog\ "}, {\" type\ ": \" cat\ "}]}" and run the test

      When using the outbox transformer with`transforms.outbox.table.expand.json.payload` set to `true` it fails to expand the JSON content in case of nested arrays of objects due to the following exception:

       

      org.apache.kafka.connect.errors.SchemaBuilderException: Cannot create field because of field name duplication @type2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at org.apache.kafka.connect.data.SchemaBuilder.field(SchemaBuilder.java:330)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.addFieldSchema(SchemaBuilderUtil.java:53)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.buildDocumentUnionSchema(SchemaBuilderUtil.java:146)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.findArrayMemberSchema(SchemaBuilderUtil.java:117)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonValueToSchema(SchemaBuilderUtil.java:78)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.addFieldSchema(SchemaBuilderUtil.java:51)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonNodeToSchemaBuilder(SchemaBuilderUtil.java:42)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonNodeToSchema(SchemaBuilderUtil.java:33)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonValueToSchema(SchemaBuilderUtil.java:80)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.addFieldSchema(SchemaBuilderUtil.java:51)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonNodeToSchemaBuilder(SchemaBuilderUtil.java:42)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.SchemaBuilderUtil.jsonNodeToSchema(SchemaBuilderUtil.java:33)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.EventRouterDelegate.apply(EventRouterDelegate.java:162)
2022-02-04T19:41:15.000Z    [Worker-0d64059776637fd2c]
at io.debezium.transforms.outbox.EventRouter.apply(EventRouter.java:25)

      It seems that the problem is caused by the fact that the code iterates over all elements of the array and it always tries to add their fields to the schema - which is fine in case of simple types, but not in the case of objects

              Unassigned Unassigned
              paw.malon@gmail.com Pawel Malon (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: