Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6028

Postgres RDS - High disk usage whenever there is a blocking transaction in database

    XMLWordPrintable

Details

    • False
    • None
    • False
    • Critical

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      Debezium connector version : 1.9.5_Final , Postgres connector

      What is the connector configuration?

      config:
          database.dbname: main
          database.hostname: HOST
          database.password: PASSWORD
          database.port: 5432
          database.server.name: server-0
          database.user: main
          decimal.handling.mode: double
          heartbeat.action.query: INSERT INTO kafka_connect_heartbeat (id, last_heartbeat_time)
            VALUES (1, NOW() at time zone 'utc') ON CONFLICT(id) DO UPDATE SET last_heartbeat_time=EXCLUDED.last_heartbeat_time;
          heartbeat.interval.ms: "10000"
          heartbeat.topics.prefix: debezium-heartbeat
          heartbeat.writeback.enabled: "true"
          heartbeat.writeback.table: public.kafka_connect_heartbeat
          plugin.name: pgoutput
          publication.autocreate.mode: filtered
          publication.name: uno_main_0
          slot.name: uno_main_0
          table.include.list: public.outbox_wal
          table.whitelist: public.outbox_wal
          tasksMax: 1
          transforms: sample
          transforms.sample.route.topic.replacement: ${routedByValue}
          transforms.sample.table.expand.json.payload: "true"
          transforms.sample.table.fields.additional.placement: type:header:eventType,id:header:messageId
          transforms.sample.type: io.debezium.transforms.outbox.EventRouter

      What is the captured database version and mode of depoyment?

      DB : AWS Postgres RDS 12.11 (Managed Service)

      What behavior do you expect?

      There should not be exponential growth in replication slot disk usage upon connector restart and message publishing should resume. 

      What behaviour do you see?

      • In case of stuck transaction(long running queries) , transaction disk usage increases exponentially.
      • Upon killing queries and restart of kafka connector , debezium catchup takes lot of time and database runs out of disk space.
      • Upon increasing additional disk space and iops approximately after 12-14 hrs connectors starts publishing messages
      • We observed restart_lsn doesn't get updated and it takes 12-14 hrs to get it matched with confirmed_flush_lsn.
      • We have implemented hearbeat as well.

      Do you see the same behaviour using the latest relesead Debezium version?

      Yes

      Do you have the connector logs, ideally from start till finish?

      No(You might be asked later to provide DEBUG/TRACE level log)

       

      How to reproduce the issue using our tutorial deployment?

       

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

      Attachments

        Activity

          People

            Unassigned Unassigned
            alokmishra1689@gmail.com Alok Mishra (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated: