Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8565

Prioritization of Live Changes During Incremental Snapshot in Debezium (PostgreSQL)

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      During the incremental snapshot process, it is critical for me to ensure that live change events are prioritized and processed near real-time. However, since my Kafka Consumer operates with a fixed consumption throughput, a challenge arises when a large number of snapshot messages are already queued. In such cases, the live change events are delayed because they are processed sequentially alongside the snapshot data. The goal is to maintain near real-time processing of live changes without being impacted by the potentially high volume of snapshot events in the queue. I have PostgreSQL tables with over 30 million entries using debezium-connector-postgres-3.0.1. Is it possible to control/delay somehow the time between the incremental chunks or approach it in another way?

      Incremental snapshot process is working properly. I just try to resolve somehow the above challenge.

      Implementation ideas (optional)

              Unassigned Unassigned
              george.tsopouridis George Tsopouridis (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: