Details

Type: Enhancement
Resolution: Done
Priority: Major
Fix Version/s: 2.4.0.Alpha1
Affects Version/s: None
Component/s: postgresql-connector
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False

SFDC Cases Counter:
SFDC Cases Links:

Description

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

With the introduction of ~~DBZ-3921~~ it's now possible to update the publication by changing the filtering in the config. The idea is to have this feature when `publication.autocreate.mode: filtered` is configured.
However, in some cases, the users have to create a publication manually and if it is already created for all tables and then the config is changed to `publication.autocreate.mode: filtered` Deebzium tries to update the publication with filters specified in the config and fails.
Instead, Debezium should emit a warning message saying that to apply filtering the publication should be recreated and proceed further.

Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

What is the connector configuration?

What is the captured database version and mode of depoyment?

(E.g. on-premises, with a specific cloud provider, etc.)

What behaviour do you expect?

Warning message is issued and replication continue working.

What behaviour do you see?

Do you see the same behaviour using the latest relesead Debezium version?

(Ideally, also verify with latest Alpha/Beta/CR version)

Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide DEBUG/TRACE level log)

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
      at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:180)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:41)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:174)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:141)
      at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
      at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.connect.errors.ConnectException: Unable to update filtered publication dbz_publication for "<redacted>"
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.createOrUpdatePublicationModeFilterted(PostgresReplicationConnection.java:214)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initPublication(PostgresReplicationConnection.java:179)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.initConnection(PostgresReplicationConnection.java:358)
      at io.debezium.connector.postgresql.connection.PostgresReplicationConnection.startStreaming(PostgresReplicationConnection.java:319)
      at io.debezium.connector.postgresql.PostgresStreamingChangeEventSource.execute(PostgresStreamingChangeEventSource.java:137)
      ... 9 more
Caused by: org.postgresql.util.PSQLException: ERROR: publication "dbz_publication" is defined as FOR ALL TABLES                                                            Detail: Tables cannot be added to or dropped from FOR ALL TABLES publications.
      at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
      at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
      at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:355)
      at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490)
      at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:329)
      at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:315)
      at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:291)
      at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:286)

How to reproduce the issue using our tutorial deployment?

Feature request or enhancement

For feature requests or enhancements, provide this information, please:

Which use case/requirement will be addressed by the proposed feature?

Implementation ideas (optional)

Attachments

Issue Links

relates to

DBZ-3921 Postgres existing publication is not updated with the new table

Closed

links to

RHEA-2024:129636 Red Hat build of Debezium 2.5.4 release

Explain failure on existing publication update when switching to `filtered` from `all_tables`