Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5915

Postgresql Data Loss on restarts

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      Bug report

      What Debezium connector do you use and what version?

      Postgresql-connector v2.0.0

      What is the connector configuration?

       

      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "value.converter": "org.apache.kafka.connect.json.JsonConverter",
      "value.converter.schemas.enable": "false",
      "key.converter.schemas.enable": "false",
      
      "database.hostname": "xxxxxxxx.rds.amazonaws.com",
      "database.password": "xxxxxxxxx",
      "database.dbname": "ledger",
      "database.user": "xxxxxxxxx",
      "database.port": "5432",
      "schema.exclude.list": "pglogical, acc, bluefin, public",
      "table.exclude.list": "ledger\\.no_cdc_.*, ledger\\.goose.*",
      
      "snapshot.mode": "never",
      "plugin.name": "pgoutput",
      "max.batch.size": "10240",
      "max.queue.size": "20480",
      "time.precision.mode": "adaptive_time_microseconds",
      "heartbeat.interval.ms": "1000",
      "topic.prefix": "postgres.ledger",
      
      "transforms": "Reroute, unwrap",
      "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
      "transforms.Reroute.topic.regex": "(.*)",
      "transforms.Reroute.topic.replacement": "cdc.$1.v3",
      "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
      "transforms.unwrap.delete.handling.mode": "rewrite",
      "transforms.unwrap.add.fields": "op,schema,table,lsn,source.ts_ms,version,snapshot"

       

       

      What is the captured database version and mode of depoyment?

      AWS RDS Postgres

      What behaviour do you expect?

      Connector should not skip any record on restarts.

      What behaviour do you see?

      Connector is sometimes skipping records on restarts.

      Do you see the same behaviour using the latest relesead Debezium version?

      Tried with 2.0.0 but looks like it would happen in latest version as well.

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

       

      *** Searching Phase ***
      2022-11-18T18:30:28.030Z Searching for WAL resume position
      2022-11-18T18:30:28.031Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} --> BEGIN
      ...
      2022-11-18T18:30:28.046Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} --> INSERT
      ...
      2022-11-18T18:30:28.068Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84059658} --> INSERT (marked as startStreamingLsn)
      ...
      2022-11-18T18:30:28.068Z WAL resume position 'LSN{8AE/84059658}' discovered
      
      *** Streaming Phase ***
      2022-11-18T18:30:28.103Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48}
      2022-11-18T18:30:28.103Z Message Type: BEGIN
      2022-11-18T18:30:28.103Z Message with LSN 'LSN{8AE/84050F48}' filtered
      2022-11-18T18:30:28.103Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} identified as already processed
      2022-11-18T18:30:28.103Z BEGIN messages are always reprocessed
      2022-11-18T18:30:28.103Z Received message at LSN LSN{8AE/84050F48}
      2022-11-18T18:30:28.103Z Message arrived from database 42000008ae84062078000290c1997430360cea399f2020
      2022-11-18T18:30:28.103Z Event: BEGIN
      2022-11-18T18:30:28.103Z Final LSN of transaction: LSN{8AE/84062078}
      2022-11-18T18:30:28.103Z Commit timestamp of transaction: 2022-11-18T18:25:31.037750Z
      2022-11-18T18:30:28.103Z XID of transaction: 216676767
      2022-11-18T18:30:28.108Z Received transactional message TransactionMessage [transactionId=216676767, commitTime=2022-11-18T18:25:31.037750Z, operation=BEGIN]
      
      
      2022-11-18T18:30:28.133Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} 
      2022-11-18T18:30:28.133Z Message Type: INSERT
      2022-11-18T18:30:28.133Z Message with LSN 'LSN{8AE/84050F48}' filtered -> This INSERT is getting skipped
      
      2022-11-18T18:30:28.141Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84059658}
      INSERT
      
      2022-11-18T18:30:28.160Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/8405DA40}
      INSERT
      
      2022-11-18T18:30:28.161Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/8405DD70}
      INSERT
      
      2022-11-18T18:30:28.163Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061A40}
      INSERT
      
      2022-11-18T18:30:28.169Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061D50}
      UPDATE
      
      2022-11-18T18:30:28.176Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061E28}
      INSERT
      
      2022-11-18T18:30:28.177Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/840620A8}
      2022-11-18T18:30:28.177Z Message Type: COMMIT
      2022-11-18T18:30:28.177Z COMMIT messages are always reprocessed
      2022-11-18T18:30:28.177Z Received message at LSN LSN{8AE/840620A8}
      2022-11-18T18:30:28.177Z Message arrived from database 4300000008ae84062078000008ae840620a8000290c1997430362020
      2022-11-18T18:30:28.177Z Event: COMMIT
      2022-11-18T18:30:28.177Z Flags: 0 (currently unused and most likely 0)
      2022-11-18T18:30:28.177Z Commit LSN: LSN{8AE/84062078}
      2022-11-18T18:30:28.177Z End LSN of transaction: LSN{8AE/840620A8}
      2022-11-18T18:30:28.177Z Commit timestamp of transaction: 2022-11-18T18:25:31.037750Z
      2022-11-18T18:30:28.177Z Received transactional message TransactionMessage [transactionId=216676767, commitTime=2022-11-18T18:25:31.037750Z, operation=COMMIT]

       

       

      How to reproduce the issue using our tutorial deployment?

      It happens sporadically across restarts.

      Attachments

        Activity

          Public project attachment banner

            context keys: [headless, issue, helper, isAsynchronousRequest, project, action, user]
            current Project key: DBZ

            People

              Unassigned Unassigned
              rdangwal Rajendra Dangwal
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: