Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
2.0.1.Final
-
None
-
False
-
None
-
False
Description
Disclaimer: It may not be a bug and maybe a missing piece in understanding of how things work with plugin `pgoutput`
When trying to use the built-in plugin `pgoutput`, I see the need to specify a `publication.name` as well a `slot.name`.
My understanding is that behind the scene,
1. the two should get connected through a `subscription`.
2. And the slot should only look at changes that are happening on the specific set of tables mentioned in the table.include.list when the setting of `publication.autocreate.mode` = filtered.
However, I am observing a very different behaviour as follows:
- No subscription getting created (update : plz see comment)
- `select * from pg_catalog.pg_stat_subscription;` is empty with 0 records
- The publication seems correct and only shows entries for the specific tables.
- However, a record creation in a table that is not part of the publication, increases the `total_txns` of the slot that was created using the connector.
- SELECT * FROM pg_stat_replication_slots;
What am i looking to achieve in general -
- I have 1 postgres database in cloudsql GCP with 1 schema and about 30 tables.
- I want to create 5 connectors, each streaming changes from 6 specific tables
- Each connector will have its own slot.name
- I want to ensure each slot is only looking into changes for tables specific to the ones defined in that connector.
- I want to verify point 4 as well that not all table changes are going into all slots.
I want to create multiple slots such that each slot only looks at X number of tables for a given same schema and database.
Bug report
What Debezium connector do you use and what version?
2.0.1
What is the connector configuration?
database.hostname: 127.0.0.1
plugin.name: pgoutput
slot.name: debezium_ps3
topic.prefix: karan_test4
database.server.name: <redacted>
database.port: 3306
database.user: <redacted>
database.password: <redacteed>
database.dbname: test_debezium_database
table.include.list: public.entries
signal.data.collection: public.signalling
publication.name: dbz_publication_3_dev
publication.autocreate.mode: filtered
What is the captured database version and mode of depoyment?
GCP (GKE) for kafkaconnect/connector
Cloudsql instance of GCP
What behaviour do you expect?
- A subscription is created that connects the publication and the slot.
- Any change in a table that is not part of the publication and hence the subscription does not affect the slot that connects the two.
What behaviour do you see?
- No subscription.
- Slots `total_txns` increases when a record is created in a table not part of the publication.
Do you see the same behaviour using the latest relesead Debezium version?
(Ideally, also verify with latest Alpha/Beta/CR version)
<Your answer>
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
<Your answer>
How to reproduce the issue using our tutorial deployment?
- Create two tables t1, t2.
- Create a connector to pull data from cloudsql instance using
- pgouput plugin
- Set a specific publication 'test'
- Set table.include.list = t1 only
- Set publication.autocreate.mode = filtered
- Set a specific slot say 'test'
- More configurations can be seen above but these were the main one.
- Start the connector
- Check publication exists (this should be fine)
- Check a subscription exits - this is an issue,
- Write a record to table t2 (and not t1)
- Run`SELECT * FROM pg_stat_replication_slots` and look at total_txns. This should not have gone up but it does.
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
<Your answer>
Implementation ideas (optional)
<Your answer>