Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4989

Outbox Transform does not allow expanded payload with additional fields in the envelope

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 1.9.1.Final, 2.0.0.Alpha1
    • 1.9.0.Final
    • outbox
    • None
    • False
    • None
    • False
    • Hide

      Note that you will need to update debezium in the example from `1.7.1.Final` to `1.9.0.Final`

      Run https://github.com/debezium/debezium-examples/tree/main/outbox with the following configuration:

      ```

      {
          "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
          "tasks.max": "1",
          "database.hostname": "order-db",
          "database.port": "5432",
          "database.user": "postgresuser",
          "database.password": "postgrespw",
          "database.dbname" : "orderdb",
          "database.server.name": "dbserver1",
          "schema.include.list": "inventory",
          "table.include.list" : "inventory.outboxevent",
          "tombstones.on.delete" : "false",
          "transforms": "outbox",
          "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
          "transforms.outbox.route.topic.replacement": "${routedByValue}.events",
          "transforms.outbox.table.field.event.timestamp": "timestamp",
          "transforms.outbox.table.fields.additional.placement": "type:envelope:eventType",
          "transforms.outbox.table.expand.json.payload": "true"
      }

      ```

      Show
      Note that you will need to update debezium in the example from `1.7.1.Final` to `1.9.0.Final` Run https://github.com/debezium/debezium-examples/tree/main/outbox with the following configuration: ``` {     "connector.class": "io.debezium.connector.postgresql.PostgresConnector",     "tasks.max": "1",     "database.hostname": "order-db",     "database.port": "5432",     "database.user": "postgresuser",     "database.password": "postgrespw",     "database.dbname" : "orderdb",     "database.server.name": "dbserver1",     "schema.include.list": "inventory",     "table.include.list" : "inventory.outboxevent",     "tombstones.on.delete" : "false",     "transforms": "outbox",     "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",     "transforms.outbox.route.topic.replacement": "${routedByValue}.events",     "transforms.outbox.table.field.event.timestamp": "timestamp",     "transforms.outbox.table.fields.additional.placement": "type:envelope:eventType",     "transforms.outbox.table.expand.json.payload": "true" } ```

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      debezium-connector-postgres version 1.9.0.Final

      What is the connector configuration?

      {
          "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
          "tasks.max": "1",
          "database.hostname": "order-db",
          "database.port": "5432",
          "database.user": "postgresuser",
          "database.password": "postgrespw",
          "database.dbname" : "orderdb",
          "database.server.name": "dbserver1",
          "schema.include.list": "inventory",
          "table.include.list" : "inventory.outboxevent",
          "tombstones.on.delete" : "false",
          "transforms": "outbox",
          "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
          "transforms.outbox.route.topic.replacement": "${routedByValue}.events",
          "transforms.outbox.table.field.event.timestamp": "timestamp",
          "transforms.outbox.table.fields.additional.placement": "type:envelope:eventType",
          "transforms.outbox.table.expand.json.payload": "true"
      }

      What is the captured database version and mode of depoyment?

      PostgreSQL 14.2, local docker

      What behaviour do you expect?

      Payload should be output in unescaped JSON along with any additional fields

      What behaviour do you see?

      ```

       

      outbox-connect-1 | Caused by: org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type STRING: class org.apache.kafka.connect.data.Struct for field: "payload"outbox-connect-1 | at org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:245)outbox-connect-1 | at org.apache.kafka.connect.data.Struct.put(Struct.java:216)outbox-connect-1 | at org.apache.kafka.connect.data.Struct.put(Struct.java:203)outbox-connect-1 | at io.debezium.transforms.outbox.EventRouterDelegate.apply(EventRouterDelegate.java:176)outbox-connect-1 | at io.debezium.transforms.outbox.EventRouter.apply(EventRouter.java:25)outbox-connect-1 | at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:50)outbox-connect-1 | at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:146)outbox-connect-1 | at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:180)outbox-connect-1 | ... 11 more

      ```{}

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      Do you have the connector logs, ideally from start till finish?

      No but can grab them if needed

      How to reproduce the issue using our tutorial deployment?

      Reproduced with https://github.com/debezium/debezium-examples/tree/main/outbox and the configuration above

              Unassigned Unassigned
              bmorganpa Brad Morgan (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: