In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

Debezium Postgres Connector 2.3.0.Final

What is the connector configuration?

class: io.debezium.connector.postgresql.PostgresConnector
tasksMax: 1
config:
database.dbname: chartofaccounts
database.hostname: chart-of-accounts-service-integration-one.ccpshfgopvpg.us-west-2.rds.amazonaws.com
database.password: ${secrets:chart-of-accounts/coa-outbox-connector:DATACLIENT_PASSWORD}
database.port: 5432
database.user: ${secrets:chart-of-accounts/coa-outbox-connector:DATACLIENT_USERNAME}
plugin.name: pgoutput
publication.autocreate.mode: filtered
schema.include.list: public
table.include.list: public.transactional_outbox
tombstones.on.delete: "false"
topic.creation.default.partitions: 10
topic.creation.default.replication.factor: 3
#required config, ignored when transforms: outbox
topic.prefix: opengov.chartofaccounts
transforms: outbox
#outbox records will be published to this topic
transforms.outbox.route.topic.replacement: opengov.chartofaccounts.v1.change
transforms.outbox.type: io.debezium.transforms.outbox.EventRouter
transforms.outbox.table.expand.json.payload: true
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
# excludes the verbose schema information
key.converter.schemas.enable: "false"
value.converter.schemas.enable: "false"
# prevent it from dropping null values
include.unknown.datatypes: "true"
table.json.payload.null.behavior: optional_bytes

What is the captured database version and mode of depoyment?

AWS Aurora PostgreSQL Serverless v2

What behaviour do you expect?

I expect that the debezium connector will publish the exact JSON payload that I write to the outbox table to Kafka. I configured it to include null values but it is still dropping values (even non-null) from my JSON payload.

What behaviour do you see?

When the message I write to my outbox table contains a list of JSON objects, values are being dropped from the JSON objects if the LAST element of the list doesn't contain that key.
For example, if my event payload contains the following list:

[
  { "code": "100", "description": "some description"},
  { "code": "200", "description": "another description"},
  { "code": "300"}
]

The event that my debezium postgres connector is publishing to kafka is dropping all of the `description` values from the list:

[
 { "code": "100"},
 { "code": "200"},
 { "code": "300"}
] 

Strangely, when it's not the last element of the list that doesn't have the key, all of the values show up as expected. For example, if the payload I'm writing to my table looks like this:

[
 { "code": "100", "description": "some description"},
 { "code": "200"},
 { "code": "300", "description": "another description"}
] 

the event published has the exact list included in the payload:

[ 
 { "code": "100", "description": "some description"}, 
 { "code": "200"},  
 { "code": "300", "description": "another description"} 
]  

Do you see the same behaviour using the latest relesead Debezium version?

verified that the behavior is the same with 2.4.0-Beta2

Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide DEBUG/TRACE level log)

yes

How to reproduce the issue using our tutorial deployment?

Write an event to the outbox table with a list of JSON objects, some that have a given key and some that don't. The issue should be reproduced if the last element of the list does not contain the key.

Feature request or enhancement

For feature requests or enhancements, provide this information, please:

Which use case/requirement will be addressed by the proposed feature?

<Your answer>

Implementation ideas (optional)

<Your answer>