Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9490

Filtered heartbeat events cause unbounded WAL growth

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.4.0.Alpha1
    • 3.1.3.Final, 3.2.2.Final, 3.3.0.Beta1
    • postgresql-connector
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      As discussed on the mailing list, a user was upgrading from Debezium 2.5 to Debezium 3.1 and was noticing that there was some behavior differences with offset management when a transformation filters an event (in their case heartbeats).

      When a `SourceRecord` goes through the transformation chain [1], the `producerRecord` is `null` and therefore `AbstractWorkerSourceTask` calls `recordDropped`. This ultimately calls `SourceTask#commitRecord`, which for Debezium is a no-op. When the `producerRecord` is not `null`, the `prepareToSendRecord` [2] method is called. This eventually calls `SourceTask#commit`, and this is where Debezium synchronizes the offset state with the replication slot.

      So if you have a period where all you have are heartbeat events and they're being filtered and not sent to Kafka, then the replication slot's confirmed LSN won't be updated, which explains your WAL growth. The reason you observe a difference in behavior appears to be related to a change in DBZ-7816 [3].

      For now, as a workaround, simply don't filter the heartbeat event and let it go the Kafka topic.

      [1]: https://github.com/apache/kafka/blob/d6133f6997e0de3931f3d98b30950aeeefd23e70/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L415-L416
      [2]: https://github.com/apache/kafka/blob/d6133f6997e0de3931f3d98b30950aeeefd23e70/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L425
      [3]: https://issues.redhat.com/browse/DBZ-7816

      In DBZ-7816 the logic was changed so that we only rely on committed offsets to the offset topic and the logic in commitRecord uses the same logic, which is the crux of why a filtered event, which only calls commitRecord and never updates the Kafka offset topic, has this issue on newer builds of Debezium.

      The present workaround is to simply not filter heartbeat events, but that is not desired or ideal in certain environments.

              Unassigned Unassigned
              ccranfor@redhat.com Chris Cranford
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: