Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-800

PostgreSQL LSNs are not committed when receiving events for filtered-out tables

XMLWordPrintable

      If a PostgreSQL table is ignored via the table.whitelist or table.blacklist connector option, Debezium ignores WAL events on that table. However, it does not commit the LSNs. This causes WAL files to build up on the database until write activity happens on the whitelisted tables. This may cause problems for tables with infrequent writes.

      From the code, it seems that LSN commits are triggered whenever the Kafka offsets are committed:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresConnectorTask.java#L136

      However, table filtering bypasses that code:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L229

      This bug might also be occurring when the table operation is unknown/ignored:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L254

              gunnar.morling Gunnar Morling
              csang1 Christopher Sang (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: