Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6307

Ensure that the connector can start from a stale timestamp more than one hour into the past


    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 2.3.0.Alpha1
    • None
    • spanner-connector
    • None
    • False
    • None
    • False

      Ensure that the connector can start from a stale timestamp more than one hour into the past.

      The issue is that we initialize the connector's schema to the start time of the connector or, if not provided, the current time: https://github.com/debezium/debezium-connector-spanner/blob/d6dfb6fb761117142ba5af9e6af61f576273aec2/src/main/java/io/debezium/connector/spanner/db/metadata/SchemaRegistry.java#L49

      This causes a problem if we want to start from an hour ago in the past, since the version retention period defaults to 1 hour maximum, and customers may not want to increase the version retention period due to cost. (https://cloud.google.com/spanner/docs/pitr). If the user tries to query the schema outside the version retention period, they will get an unretriable error.

      It seems to me that the solution we can go with is this:

      (1) Upon startup, we should initialize the schema to be read at a timestamp which is the max (connector start time, current time) (2) When we detect inconsistencies in the schema and try to update the schema, we should first do a schema lookup from the INFORMATION_SCHEMA.

      (3) If we receive an error indicating that we tried to do a lookup outside of the version retention period, we should then simply add all new or modified columns that we detected in the data change record to the connector's view of the schema.

            nancyxu567@gmail.com Nancy Xu (Inactive)
            nancyxu567@gmail.com Nancy Xu (Inactive)
            0 Vote for this issue
            3 Start watching this issue