-
Bug
-
Resolution: Done
-
Major
-
2.5.2.Final
-
None
-
False
-
None
-
False
What Debezium connector do you use and what version?
PostgreSQL 2.5.2.Final
What is the connector configuration?
{ "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "errors.log.include.messages": "true", "reselector.reselect.columns.include.list": "public.str2665_current:hstorecol", "slot.name": "streamkap_pgoutput_slot", "publication.name": "streamkap_pub", "signal.enabled.channels": "source", "reselector.reselect.unavailable.values": "true", "topic.prefix": "STR2665_Current", "reselector.reselect.use.event.key": "false", "signal.data.collection": "public.streamkap_signal", "value.converter": "io.confluent.connect.avro.AvroConverter", "errors.log.enable": "true", "key.converter": "io.confluent.connect.avro.AvroConverter", "reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor", "database.user": "local", "database.dbname": "localdb", "post.processors": "reselector", "heartbeat.interval.ms": "60000", "database.port": "5432", "plugin.name": "pgoutput", "value.converter.schema.registry.url": "http://schema-registry:8081", "database.hostname": "postgresql", "database.password": "******", "schema.name.adjustment.mode": "avro", "name": "STR2665_Current", "table.include.list": "public.str2665_current", "key.converter.schema.registry.url": "http://schema-registry:8081", "reselector.reselect.null.values": "false", "snapshot.mode": "never" }
What is the captured database version and mode of depoyment?
PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2), local Docker deployment
What behaviour do you expect?
The Re-select post processor to ignore snapshot window events
What behaviour do you see?
[2024-03-06 17:58:49,423] TRACE Received change record Struct{after=Struct{id=1124e905-2c43-41c1-abf3-f1562c628d01-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929401,db=localdb,sequence=["27685160","27685160"],schema=public,table=streamkap_signal,txId=781,lsn=27685160},op=c,ts_ms=1709747929408} for CREATE operation on key Struct{id=1124e905-2c43-41c1-abf3-f1562c628d01-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67128}, txId=781, messageType=INSERT, lastCommitLsn=LSN{0/1A67128}, timestamp=2024-03-06T17:58:49.420910Z, snapshot=FALSE, schema=public, table=str2665_current], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67128}, lastCommitLsn=LSN{0/1A67128}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher) [2024-03-06 17:58:49,423] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor) [2024-03-06 17:58:49,424] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open,type=snapshot-window-open},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929422,db=localdb,sequence=["27685432","27685432"],schema=public,table=streamkap_signal,txId=782,lsn=27685432},op=c,ts_ms=1709747929424} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67238}, txId=782, messageType=INSERT, lastCommitLsn=LSN{0/1A67238}, timestamp=2024-03-06T17:58:49.422007Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67238}, lastCommitLsn=LSN{0/1A67238}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher) [2024-03-06 17:58:49,424] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open,type=snapshot-window-open},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929422,db=localdb,sequence=["27685432","27685432"],schema=public,table=streamkap_signal,txId=782,lsn=27685432},op=c,ts_ms=1709747929424} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67238}, txId=782, messageType=INSERT, lastCommitLsn=LSN{0/1A67238}, timestamp=2024-03-06T17:58:49.422007Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67238}, lastCommitLsn=LSN{0/1A67238}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=true, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher) [2024-03-06 17:58:49,424] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor) [2024-03-06 17:58:49,425] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929423,db=localdb,sequence=["27685704","27685704"],schema=public,table=streamkap_signal,txId=783,lsn=27685704},op=c,ts_ms=1709747929425} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67348}, txId=783, messageType=INSERT, lastCommitLsn=LSN{0/1A67348}, timestamp=2024-03-06T17:58:49.423398Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67348}, lastCommitLsn=LSN{0/1A67348}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=true, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher) [2024-03-06 17:58:49,425] INFO Skipping read chunk because snapshot is not running (io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource) [2024-03-06 17:58:49,425] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929423,db=localdb,sequence=["27685704","27685704"],schema=public,table=streamkap_signal,txId=783,lsn=27685704},op=c,ts_ms=1709747929425} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67348}, txId=783, messageType=INSERT, lastCommitLsn=LSN{0/1A67348}, timestamp=2024-03-06T17:58:49.423398Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67348}, lastCommitLsn=LSN{0/1A67348}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher) [2024-03-06 17:58:49,425] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor)
Window open and close events are being processed by the Re-select post processor even though the signal table's not included in the table.include.list.
Do you have the connector logs, ideally from start till finish?
Yes, please see attached: 202403061800.log
- links to
-
RHEA-2024:129636 Red Hat build of Debezium 2.5.4 release