In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

 
debezium-connector-postgresql:1.9.3

What is the connector configuration?

I use terraform to create my connector so config syntax might look a bit weird:

config =

{ 
  "name" = "test_db_outbox_connector",
  "connector.class" = "io.debezium.connector.postgresql.PostgresConnector",
  "errors.log.enable" = "true",
  "errors.log.include.messages" = "true",
  "errors.retry.timeout" = "300000",
  "errors.retry.delay.max.ms" = "60000",
  "snapshot.mode" = "initial",
  "plugin.name" = "pgoutput",
  "tasks.max" = "1",
  "database.hostname" = local.test_db.host,

"database.port" = local.test_db.port,

"database.user" = local.test_db.username,

"database.dbname" = local.test_db.db_name,

"database.server.name" = "test_db_outbox",
  "slot.name" = "outbox_connector_slot",
  "key.converter" = "org.apache.kafka.connect.storage.StringConverter",
  "key.converter.schemas.enable" = "false",
  "value.converter" = "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schema.registry.url" = local.schema_registry_url,

"value.converter.schemas.enable" = "false",
  "value.converter.auto.register.schemas" = "false",
  "include.schema.changes" = "false",
  "table.include.list" = "public.outbox",
  "publication.autocreate.mode" = "disabled",
  "publication.name" = "outbox_publication",
  "poll.interval.ms" = "500",
  "slot.max.retries" = "3",
  "slot.retry.delay.ms" = "5000",
  "status.update.interval.ms" = "5000",
  "heartbeat.interval.ms" = "3000",
  "heartbeat.topics.prefix" = "debezium-heartbeat",
  "transforms" = "outbox",
  "transforms.outbox.type" = "io.debezium.transforms.outbox.EventRouter",
  "transforms.outbox.table.fields.additional.placement" = "aggregateid:header:AggregateId,generated_at:header:GeneratedAt,id:header:Id,origin:header:Origin",
  "transforms.outbox.table.field.event.id" = "id",
  "transforms.outbox.table.field.event.key" = "aggregateid",
  "transforms.outbox.table.field.event.payload" = "payload",
  "transforms.outbox.table.expand.json.payload" = "true",
  "transforms.outbox.route.by.field" = "event_name",
  "transforms.outbox.route.topic.replacement" = "test_db.public.outbox" 
}

config_sensitive =

{ "database.password" = local.test_db.password }

 

What is the captured database version and mode of depoyment?

Deployed via AWS RDS. postgres 12.11

What behaviour do you expect?

When `transforms.outbox.table.expand.json.payload = true` and then when I insert `'{"heartbeat": 1, "email": null}'` I expect in kafka to get `{"heartbeat": 1, "email: null}`

What behaviour do you see?

When `transforms.outbox.table.expand.json.payload = true` and then I insert `'{"heartbeat": 1, "email": null}'` in kafka I get `{"heartbeat": 1}`.

When `transforms.outbox.table.expand.json.payload = false` I do correctly get `"{\"email\": null, \"heartbeat\": 1}"`

 

Do you see the same behaviour using the latest relesead Debezium version?

I haven't tried the behavior on debezium 2.0 because it is a major release bump.

Also in the github repo there are no new commits within the outbox folder when I toggle between 2.0 and 1.9

https://github.com/debezium/debezium/tree/2.0/debezium-core/src/main/java/io/debezium/transforms/outbox

https://github.com/debezium/debezium/tree/1.9/debezium-core/src/main/java/io/debezium/transforms/outbox

I think the bug might be within this section:

https://github.com/debezium/debezium/blob/4418722d2e39c2819caaa7858dfd8a23bb730b7b/debezium-core/src/main/java/io/debezium/transforms/outbox/EventRouterDelegate.java#L159-L165

Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide DEBUG/TRACE level log)

N/A