Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6018

subscription not created when plugin is pgoutput

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Major
    • None
    • 2.0.1.Final
    • postgresql-connector
    • None
    • False
    • None
    • False

    Description

      Disclaimer: It may not be a bug and maybe a missing piece in understanding of how things work with plugin `pgoutput`

      When trying to use the built-in plugin `pgoutput`, I see the need to specify a `publication.name` as well a `slot.name`. 

      My understanding is that behind the scene,

      1. the two should get connected through a `subscription`.

      2. And the slot should only look at changes that are happening on the specific set of tables mentioned in the table.include.list when the setting of `publication.autocreate.mode` = filtered. 

       

      However, I am observing a very different behaviour as follows:

      1. No subscription getting created (update : plz see comment) 
        1. `select * from pg_catalog.pg_stat_subscription;` is empty with 0 records
      2. The publication seems correct and only shows entries for the specific tables.
      3. However, a record creation in a table that is not part of the publication, increases the `total_txns` of the slot that was created using the connector.
        1. SELECT * FROM pg_stat_replication_slots;

      What am i looking to achieve in general -

      1. I have 1 postgres database in cloudsql GCP with 1 schema and about 30 tables.
      2. I want to create 5 connectors, each streaming changes from 6 specific tables 
      3. Each connector will have its own slot.name
      4. I want to ensure each slot is only looking into changes for tables specific to the ones defined in that connector.
      5. I want to verify point 4 as well that not all table changes are going into all slots. 

      I want to create multiple slots such that each slot only looks at X number of tables for a given same schema and database.

      Bug report

      What Debezium connector do you use and what version?

      2.0.1

      What is the connector configuration?

      database.hostname: 127.0.0.1
      plugin.name: pgoutput
      slot.name: debezium_ps3
      topic.prefix: karan_test4
      database.server.name: <redacted>
      database.port: 3306
      database.user: <redacted>
      database.password: <redacteed>
      database.dbname: test_debezium_database
      table.include.list: public.entries
      signal.data.collection: public.signalling
      publication.name: dbz_publication_3_dev
      publication.autocreate.mode: filtered
       

      What is the captured database version and mode of depoyment?

      GCP (GKE) for kafkaconnect/connector

      Cloudsql instance of GCP

      What behaviour do you expect?

      1. A subscription is created that connects the publication and the slot.
      2. Any change in a table that is not part of the publication and hence the subscription does not affect the slot that connects the two. 

      What behaviour do you see?

      1. No subscription.
      2. Slots `total_txns` increases when a record is created in a table not part of the publication.

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      <Your answer>

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      <Your answer>

      How to reproduce the issue using our tutorial deployment?

      1. Create two tables t1, t2.
      2. Create a connector to pull data from cloudsql instance using
        1. pgouput plugin
        2. Set a specific publication 'test'
        3. Set table.include.list = t1 only
        4. Set publication.autocreate.mode = filtered
        5. Set a specific slot say 'test'
      3. More configurations can be seen above but these were the main one.
      4. Start the connector
        1. Check publication exists (this should be fine)
        2. Check a subscription exits - this is an issue,
      5. Write a record to table t2 (and not t1)
        1. Run`SELECT * FROM pg_stat_replication_slots` and look at total_txns. This should not have gone up  but it does.

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

      Attachments

        Activity

          People

            Unassigned Unassigned
            ee07dazn Karan Rewari (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: