Details
-
Bug
-
Resolution: Done
-
Major
-
2.0.0.Final
-
None
-
False
-
None
-
False
Description
Bug report
What Debezium connector do you use and what version?
Postgresql-connector v2.0.0
What is the connector configuration?
"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "key.converter.schemas.enable": "false", "database.hostname": "xxxxxxxx.rds.amazonaws.com", "database.password": "xxxxxxxxx", "database.dbname": "ledger", "database.user": "xxxxxxxxx", "database.port": "5432", "schema.exclude.list": "pglogical, acc, bluefin, public", "table.exclude.list": "ledger\\.no_cdc_.*, ledger\\.goose.*", "snapshot.mode": "never", "plugin.name": "pgoutput", "max.batch.size": "10240", "max.queue.size": "20480", "time.precision.mode": "adaptive_time_microseconds", "heartbeat.interval.ms": "1000", "topic.prefix": "postgres.ledger", "transforms": "Reroute, unwrap", "transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter", "transforms.Reroute.topic.regex": "(.*)", "transforms.Reroute.topic.replacement": "cdc.$1.v3", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.delete.handling.mode": "rewrite", "transforms.unwrap.add.fields": "op,schema,table,lsn,source.ts_ms,version,snapshot"
What is the captured database version and mode of depoyment?
AWS RDS Postgres
What behaviour do you expect?
Connector should not skip any record on restarts.
What behaviour do you see?
Connector is sometimes skipping records on restarts.
Do you see the same behaviour using the latest relesead Debezium version?
Tried with 2.0.0 but looks like it would happen in latest version as well.
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
*** Searching Phase *** 2022-11-18T18:30:28.030Z Searching for WAL resume position 2022-11-18T18:30:28.031Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} --> BEGIN ... 2022-11-18T18:30:28.046Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} --> INSERT ... 2022-11-18T18:30:28.068Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84059658} --> INSERT (marked as startStreamingLsn) ... 2022-11-18T18:30:28.068Z WAL resume position 'LSN{8AE/84059658}' discovered *** Streaming Phase *** 2022-11-18T18:30:28.103Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} 2022-11-18T18:30:28.103Z Message Type: BEGIN 2022-11-18T18:30:28.103Z Message with LSN 'LSN{8AE/84050F48}' filtered 2022-11-18T18:30:28.103Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} identified as already processed 2022-11-18T18:30:28.103Z BEGIN messages are always reprocessed 2022-11-18T18:30:28.103Z Received message at LSN LSN{8AE/84050F48} 2022-11-18T18:30:28.103Z Message arrived from database 42000008ae84062078000290c1997430360cea399f2020 2022-11-18T18:30:28.103Z Event: BEGIN 2022-11-18T18:30:28.103Z Final LSN of transaction: LSN{8AE/84062078} 2022-11-18T18:30:28.103Z Commit timestamp of transaction: 2022-11-18T18:25:31.037750Z 2022-11-18T18:30:28.103Z XID of transaction: 216676767 2022-11-18T18:30:28.108Z Received transactional message TransactionMessage [transactionId=216676767, commitTime=2022-11-18T18:25:31.037750Z, operation=BEGIN] 2022-11-18T18:30:28.133Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84050F48} 2022-11-18T18:30:28.133Z Message Type: INSERT 2022-11-18T18:30:28.133Z Message with LSN 'LSN{8AE/84050F48}' filtered -> This INSERT is getting skipped 2022-11-18T18:30:28.141Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84059658} INSERT 2022-11-18T18:30:28.160Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/8405DA40} INSERT 2022-11-18T18:30:28.161Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/8405DD70} INSERT 2022-11-18T18:30:28.163Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061A40} INSERT 2022-11-18T18:30:28.169Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061D50} UPDATE 2022-11-18T18:30:28.176Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/84061E28} INSERT 2022-11-18T18:30:28.177Z Streaming requested from LSN LSN{8AE/84050F48}, received LSN LSN{8AE/840620A8} 2022-11-18T18:30:28.177Z Message Type: COMMIT 2022-11-18T18:30:28.177Z COMMIT messages are always reprocessed 2022-11-18T18:30:28.177Z Received message at LSN LSN{8AE/840620A8} 2022-11-18T18:30:28.177Z Message arrived from database 4300000008ae84062078000008ae840620a8000290c1997430362020 2022-11-18T18:30:28.177Z Event: COMMIT 2022-11-18T18:30:28.177Z Flags: 0 (currently unused and most likely 0) 2022-11-18T18:30:28.177Z Commit LSN: LSN{8AE/84062078} 2022-11-18T18:30:28.177Z End LSN of transaction: LSN{8AE/840620A8} 2022-11-18T18:30:28.177Z Commit timestamp of transaction: 2022-11-18T18:25:31.037750Z 2022-11-18T18:30:28.177Z Received transactional message TransactionMessage [transactionId=216676767, commitTime=2022-11-18T18:25:31.037750Z, operation=COMMIT]
How to reproduce the issue using our tutorial deployment?
It happens sporadically across restarts.
