Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7611

Re-select post processor performance

XMLWordPrintable

      What Debezium connector do you use and what version?

      PostgreSQL 2.5.2.Final

      What is the connector configuration?

      {
      	"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      	"errors.log.include.messages": "true",
      	"reselector.reselect.columns.include.list": "public.str2665_current:hstorecol",
      	"slot.name": "streamkap_pgoutput_slot",
      	"publication.name": "streamkap_pub",
      	"signal.enabled.channels": "source",
      	"reselector.reselect.unavailable.values": "true",
      	"topic.prefix": "STR2665_Current",
      	"reselector.reselect.use.event.key": "false",
      	"signal.data.collection": "public.streamkap_signal",
      	"value.converter": "io.confluent.connect.avro.AvroConverter",
      	"errors.log.enable": "true",
      	"key.converter": "io.confluent.connect.avro.AvroConverter",
      	"reselector.type": "io.debezium.processors.reselect.ReselectColumnsPostProcessor",
      	"database.user": "local",
      	"database.dbname": "localdb",
      	"post.processors": "reselector",
      	"heartbeat.interval.ms": "60000",
      	"database.port": "5432",
      	"plugin.name": "pgoutput",
      	"value.converter.schema.registry.url": "http://schema-registry:8081",
      	"database.hostname": "postgresql",
      	"database.password": "******",
      	"schema.name.adjustment.mode": "avro",
      	"name": "STR2665_Current",
      	"table.include.list": "public.str2665_current",
      	"key.converter.schema.registry.url": "http://schema-registry:8081",
      	"reselector.reselect.null.values": "false",
      	"snapshot.mode": "never"
      }
      

      What is the captured database version and mode of depoyment?

      PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2), local Docker deployment

      What behaviour do you expect?

      The Re-select post processor to ignore snapshot window events

      What behaviour do you see?

      [2024-03-06 17:58:49,423] TRACE Received change record Struct{after=Struct{id=1124e905-2c43-41c1-abf3-f1562c628d01-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929401,db=localdb,sequence=["27685160","27685160"],schema=public,table=streamkap_signal,txId=781,lsn=27685160},op=c,ts_ms=1709747929408} for CREATE operation on key Struct{id=1124e905-2c43-41c1-abf3-f1562c628d01-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67128}, txId=781, messageType=INSERT, lastCommitLsn=LSN{0/1A67128}, timestamp=2024-03-06T17:58:49.420910Z, snapshot=FALSE, schema=public, table=str2665_current], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67128}, lastCommitLsn=LSN{0/1A67128}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher)
      [2024-03-06 17:58:49,423] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor)
      [2024-03-06 17:58:49,424] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open,type=snapshot-window-open},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929422,db=localdb,sequence=["27685432","27685432"],schema=public,table=streamkap_signal,txId=782,lsn=27685432},op=c,ts_ms=1709747929424} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67238}, txId=782, messageType=INSERT, lastCommitLsn=LSN{0/1A67238}, timestamp=2024-03-06T17:58:49.422007Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67238}, lastCommitLsn=LSN{0/1A67238}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher)
      [2024-03-06 17:58:49,424] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open,type=snapshot-window-open},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929422,db=localdb,sequence=["27685432","27685432"],schema=public,table=streamkap_signal,txId=782,lsn=27685432},op=c,ts_ms=1709747929424} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-open} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67238}, txId=782, messageType=INSERT, lastCommitLsn=LSN{0/1A67238}, timestamp=2024-03-06T17:58:49.422007Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67238}, lastCommitLsn=LSN{0/1A67238}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=true, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher)
      [2024-03-06 17:58:49,424] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor)
      [2024-03-06 17:58:49,425] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929423,db=localdb,sequence=["27685704","27685704"],schema=public,table=streamkap_signal,txId=783,lsn=27685704},op=c,ts_ms=1709747929425} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67348}, txId=783, messageType=INSERT, lastCommitLsn=LSN{0/1A67348}, timestamp=2024-03-06T17:58:49.423398Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67348}, lastCommitLsn=LSN{0/1A67348}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=true, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher)
      [2024-03-06 17:58:49,425] INFO Skipping read chunk because snapshot is not running (io.debezium.pipeline.source.snapshot.incremental.AbstractIncrementalSnapshotChangeEventSource)
      [2024-03-06 17:58:49,425] TRACE Received change record Struct{after=Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close,type=snapshot-window-close},source=Struct{version=2.5.2.Final,connector=postgresql,name=STR2665_Current,ts_ms=1709747929423,db=localdb,sequence=["27685704","27685704"],schema=public,table=streamkap_signal,txId=783,lsn=27685704},op=c,ts_ms=1709747929425} for CREATE operation on key Struct{id=b146d879-d73f-4c42-9065-90cb44608cf2-close} with context PostgresOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.postgresql.Source:STRUCT}, sourceInfo=source_info[server='STR2665_Current'db='localdb', lsn=LSN{0/1A67348}, txId=783, messageType=INSERT, lastCommitLsn=LSN{0/1A67348}, timestamp=2024-03-06T17:58:49.423398Z, snapshot=FALSE, schema=public, table=streamkap_signal], lastSnapshotRecord=false, lastCompletelyProcessedLsn=LSN{0/1A67348}, lastCommitLsn=LSN{0/1A67348}, streamingStoppingLsn=null, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], incrementalSnapshotContext=IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]] (io.debezium.pipeline.EventDispatcher)
      [2024-03-06 17:58:49,425] DEBUG No columns require re-selection. (io.debezium.processors.reselect.ReselectColumnsPostProcessor)
      

      Window open and close events are being processed by the Re-select post processor even though the signal table's not included in the table.include.list.

      Do you have the connector logs, ideally from start till finish?

      Yes, please see attached: 202403061800.log

              ccranfor@redhat.com Chris Cranford
              jonathono Jonathon Ogden
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: