Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-800

PostgreSQL LSNs are not committed when receiving events for filtered-out tables

    XMLWordPrintable

    Details

      Description

      If a PostgreSQL table is ignored via the table.whitelist or table.blacklist connector option, Debezium ignores WAL events on that table. However, it does not commit the LSNs. This causes WAL files to build up on the database until write activity happens on the whitelisted tables. This may cause problems for tables with infrequent writes.

      From the code, it seems that LSN commits are triggered whenever the Kafka offsets are committed:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresConnectorTask.java#L136

      However, table filtering bypasses that code:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L229

      This bug might also be occurring when the table operation is unknown/ignored:
      https://github.com/debezium/debezium/blob/v0.8.0.Final/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/RecordsStreamProducer.java#L254

        Attachments

          Activity

            People

            Assignee:
            gunnar.morling Gunnar Morling
            Reporter:
            csang Christopher Sang (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: