Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-379

Postgres connectors stops to work after concurrent schema changes and updates

    XMLWordPrintable

Details

    Description

      I'm exploring how the PG connector works in case of concurrent schema changes and updates and it seems the connector stops to work in the following scenario:

      • Set a break point to PostgresSchema#refresh()
      • Run this DDL: alter table customers add column middle_name varchar(255);
      • Run this DML: update customers set first_name='Mary' where id = 1004; this will hit the break point set above. Wait there.
      • Run this DDL: alter table customers add column another_middle_name varchar(255);
      • Continue after the break point

      There will be no update for the DML statement in the Kafka topic. I reckon the connector runs into some kind of error, it doesn't log anything, though.

      I suspect the sequence above poses an issue either way: it seems the schema-update upon processing the DML would read the second DDL which isn't valid yet at this point, though.

      Attachments

        Issue Links

          is blocked by

          Feature Request - Feature requests from customers and/or users DBZ-453 Support new wal2json type specifiers

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-485 Protobuf message should contain type modifiers

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-486 Protobuf message should contain optional flags

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-487 Protobuf message should contain primary key information

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Open
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. DBZ-490 postgresql-connector does not detect column type changes

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is related to

          Bug - A problem that impairs or prevents the functions of the product. DBZ-454 Postgres wal2json drops unknown types during updates

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. DBZ-512 For old connector OID should be used to detect schema change

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              jpechane Jiri Pechanec
              gunnar.morling Gunnar Morling
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: