Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5401

Networking issue between debezium and postgresql

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Blocker
    • None
    • 1.9.2.Final
    • postgresql-connector
    • None
    • False
    • None
    • False
    • Hide
      1. Have a Amazon EKS environment ready
      2. deploy the postgres serivce managed by repmgr.
      3. deploy the kafka cluster managed by Strimzi API.
      4. deploy the kafka connect and kafka connector(debezium) managed by Strimzi API.
      Show
      Have a Amazon EKS environment ready deploy the postgres serivce managed by repmgr. deploy the kafka cluster managed by Strimzi API. deploy the kafka connect and kafka connector(debezium) managed by Strimzi API.

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      io.debezium.connector.postgresql.PostgresConnector

      version 1.9.2 Final

      What is the connector configuration?

      tasks.max: 1
      plugin.name: pgoutput
      database.sslmode: "require"
      database.server.name: publish
      database.hostname: "postgresql-ha-pgpool"

      database.port: 5432
      database.user: "${directory:/opt/kafka/external-configuration/connector-config-postgresql-username:postgresql-username}"
      database.password: "${directory:/opt/kafka/external-configuration/connector-config-postgresql-password:postgresql-password}"
      database.dbname: mydb
      table.include.list: "public.recommendation_p([0-9]+)"
      column.include.list: "public.recommendation_p([0-9]).status,public.recommendation_p([0-9]).path,public.mytable_p([0-9]+).site_id"
      message.key.columns: "public.recommendation_p([0-9]+):site_id"
      transforms: Reroute,filter,unwrap
      transforms.Reroute.type: io.debezium.transforms.ByLogicalTableRouter
      transforms.Reroute.topic.regex: publish.public.recommendation_p([0-9]+)
      transforms.Reroute.topic.replacement: publish.public.recommendation
      transforms.Reroute.key.enforce.uniqueness: false
      transforms.filter.type: io.debezium.transforms.Filter
      transforms.filter.language: jsr223.groovy
      transforms.filter.condition: value.op == 'u' && value.after.status == 'PUBLISH_INITIATED'
      transforms.filter.topic.regex: publish.public.recommendation
      transforms.unwrap.type: io.debezium.transforms.ExtractNewRecordState

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      Postgres 13.4

      It is deployed together with repmgr and pgpool on the Amazon EKS cluster for high availability.

      What behaviour do you expect?

      It's very low-latency from creating the data change events in the postgres to getting messages from the kafka topic. Also, there shouldn't be any errors in the logs of debezium.

      What behaviour do you see?

      We see there are some networking issue logs between debezium and postgres. Seems like debezium are reading from/writing to the other end which is already closed. The error looks similar to this issue, but it said it was fixed already in the earlier version. Also, the performance of debezium is not consistent, sometimes it takes few seconds to receive the messages from kafka, sometimes it takes up to several minutes for the consumer to receive. We are wondering whether this behavior is expected or not.

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      We only use 1.9.2 Final version.

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      The connector log is attached below. Basically the log messages in the attachement is repeated as the debezium is running.

      How to reproduce the issue using our tutorial deployment?

      Our debezium, postgres and kafka cluster are all deployed on the Amazon EKS environment, the tutorial here might not reproduce the issue.

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

      Attachments

        Activity

          People

            Unassigned Unassigned
            yweiming Sabonis Yang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: