Details
-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
1.4.2.Final
-
None
-
False
-
False
-
Undefined
-
Description
Marked as a bug, but depends on what configuration means.
We are using AWS RDS as a Postgres database based on Postgres 11 version.
We are using Debezium 1.4.1 version with pgoutput and outbox router.
Recently, I've been testing what will happen if the topic will not be configured in Kafka.
Normally, in that case, connector login warns & stuck in an infinite loop as the task is unable to produce an event to Kafka:
WARN [Producer clientId=connector-producer-piotrek-test-3.outbox_events_debezium_connector-0] Error while fetching metadata with correlation id 3884 : {piotrek-test-123.partners.feature-flag-events=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
So, in that case, even when I set up event.processing.failure.handling.mode still raises the errors and is stuck in the loop forever.
As an outbox router is configured to be able to fetch data from one table and route them to multiple different topics, one typo can block the entire queue from consuming & bloat databases as the connector will be stuck on a single event forever/until someone creates a topic.
Connector configuration looks
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "slot.name": "debezium",
"tasks.max": "4",
"publication.name": "outbox_publication",
"transforms": "outbox",
"tombstones.on.delete": "false", "transforms.outbox.route.topic.replacement": "piotrek-test-3.${routedByValue}",
"publication.autocreate.mode": "disabled",
"database.user": "XXXX",
"database.dbname": "XXXX", "transforms.outbox.table.fields.additional.placement": "uuid:envelope:uuid,event_type:envelope:event_type,timestamp:envelope:timestamp",
"transforms.outbox.table.field.event.key": "partition_key", "transforms.outbox.table.field.event.timestamp": "timestamp", "database.server.name": "XXXX",
"transforms.outbox.table.field.event.id": "uuid", "transforms.outbox.route.by.field": "topic_name", "event.processing.failure.handling.mode": "skip",
"database.port": "5432",
"plugin.name": "pgoutput",
"topic.creation.enable": "false",
"database.hostname": "XXXX",
"database.password": "XXXX",
"name": "outbox_events_debezium_connector", "transforms.outbox.table.field.event.payload": "proto_payload", "table.include.list": "public.outbox_events",
"snapshot.mode": "exported", "transforms.outbox.table.field.event.payload.id": "partition_key"
}
Attachments
Issue Links
- links to
