Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9427

Add guardrails to prevent OOM due to too many tables configured

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Unresolved
    • Icon: Major Major
    • 3.4.0.Alpha1
    • None
    • core-library
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      Any Debezium connector that captures changes from databases with many tables (e.g., Postgres, MySQL, SQL Server).

      What is the connector configuration?

      Configured 28K tables with large number of columns for MySql connector and it was taking more than 5 GB heap.

      What behavior do you expect?

      The connector should validate the number of tables being captured against a safe limit and either:

      • fail fast with a clear error, or
      • log a warning (based on configuration).

      This prevents misconfigurations from exhausting heap memory.

      What behavior do you see?

      When a connector is configured with too many tables, internal in-memory structures grow excessively, eventually leading to OOM and the connect worker crashes.

      Do you see the same behaviour using the latest released Debezium version?

      Yes. The current Debezium release does not include any guardrail check for the number of tables configured.

      How to reproduce the issue using our tutorial deployment?

      • Create large number of tables with large number of columns on the DB.
      • Configure the connector with a regex (e.g., table.include.list=.*) that captures all of the tables.
      • Launch the connector.
      • Observe the heap usage grow abnormally, leading to eventual OOM.

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      Prevent connector instability by introducing guardrail checks for the number of tables configured. This ensures misconfigurations or malicious setups don't cause memory exhaustion and downtime. It safeguards:

      • End users from accidental crashes, and
      • Vendors/managed services from infrastructure risks.

      Implementation ideas (optional)

      • Introduce a configuration such as:
        • guardrail.tables.max → disabled by default (0).
        • guardrail.limit.actionfail|warn (default: warn).
      • Apply the guardrail check at connector startup and optionally at runtime (when new tables are discovered).
      • Emit clear logs or fail the connector early depending on the configured action.

              Unassigned Unassigned
              rdangwal Rajendra Dangwal
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: