Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-512

For old connector OID should be used to detect schema change

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 0.7.1
    • 0.7.0
    • postgresql-connector
    • None
    • Hide

      When the postgis extension is created, PG sets up the `spatial_ref_sys` table with 5500 rows.

      For tests I want to ignore it, so https://github.com/debezium/debezium/pull/374/files#diff-062e7af96d2f14c29aff466562a13e9aR122 waits for <20s for those records to load (into `postgis.spatial_ref_sys`, so dropped by the filter), before continuing on with testing actual values in `public.postgis_table`.

      This is now taking 5+ minutes (Eclipse debug).

      Appears to happen with both wal2json and decoderbufs. The above timings are from wal2json though.

      Show
      When the postgis extension is created, PG sets up the `spatial_ref_sys` table with 5500 rows. For tests I want to ignore it, so https://github.com/debezium/debezium/pull/374/files#diff-062e7af96d2f14c29aff466562a13e9aR122 waits for <20s for those records to load (into `postgis.spatial_ref_sys`, so dropped by the filter), before continuing on with testing actual values in `public.postgis_table`. This is now taking 5+ minutes (Eclipse debug). Appears to happen with both wal2json and decoderbufs. The above timings are from wal2json though.

      I think the changes for DBZ-379 is doing some bad things. Like querying the database every received message for updated type information...

      2017-12-15 13:03:33,240 DEBUG  Postgres|test_server|records-stream-producer  received new message at position 0/EBB2ACF
io.debezium.connector.postgresql.connection.wal2json.Wal2JsonReplicationMessage@1e753d69   [io.debezium.connector.postgresql.RecordsStreamProducer]
2017-12-15 13:03:33,240 DEBUG  Postgres|test_server|records-stream-producer  detected new type for column 'srid', old type was '4', new type is '23'; refreshing table schema   [io.debezium.connector.postgresql.RecordsStreamProducer]
2017-12-15 13:03:33,267 DEBUG  Postgres|test_server|records-stream-producer  Connected to jdbc:postgresql://localhost:5432/postgres with {sslmode=disable, user=postgres, password=postgres, server.name=test_server, assumeMinServerVersion=9.4}   [io.debezium.jdbc.JdbcConnection]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  refreshing DB schema for table 'postgis.spatial_ref_sys'   [io.debezium.connector.postgresql.PostgresSchema]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapping table 'postgis.spatial_ref_sys' to schemas under 'test_server.postgis.spatial_ref_sys'   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srid' (INT32) from column srid int4(10,0) NOT NULL   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srid' (INT32) from column srid int4(10,0) NOT NULL   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'auth_name' (STRING) from column auth_name varchar(256,0)   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'auth_srid' (INT32) from column auth_srid int4(10,0)   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'srtext' (STRING) from column srtext varchar(2048,0)   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  - field 'proj4text' (STRING) from column proj4text varchar(2048,0)   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapped primary key for table 'postgis.spatial_ref_sys' to schema: {"name" : "test_server.postgis.spatial_ref_sys.Key", "type" : "STRUCT", "optional" : "false", "fields" : [{"name" : "srid", "index" : "0", "schema" : {"type" : "INT32", "optional" : "false"}}]}   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  Mapped columns for table 'postgis.spatial_ref_sys' to schema: {"name" : "test_server.postgis.spatial_ref_sys.Value", "type" : "STRUCT", "optional" : "true", "fields" : [{"name" : "srid", "index" : "0", "schema" : {"type" : "INT32", "optional" : "false"}}, {"name" : "auth_name", "index" : "1", "schema" : {"type" : "STRING", "optional" : "true"}}, {"name" : "auth_srid", "index" : "2", "schema" : {"type" : "INT32", "optional" : "true"}}, {"name" : "srtext", "index" : "3", "schema" : {"type" : "STRING", "optional" : "true"}}, {"name" : "proj4text", "index" : "4", "schema" : {"type" : "STRING", "optional" : "true"}}]}   [io.debezium.relational.TableSchemaBuilder]
2017-12-15 13:03:33,313 DEBUG  Postgres|test_server|records-stream-producer  sending create event 'SourceRecord{sourcePartition={server=test_server}, sourceOffset={lsn=247147215, txId=1152, ts_usec=1513342952883563000}} ConnectRecord{topic='test_server.postgis.spatial_ref_sys', kafkaPartition=null, key=Struct{srid=2113}, value=Struct{after=Struct{srid=2113,auth_name=EPSG,auth_srid=2113,srtext=PROJCS["NZGD2000...

      I guess the key bit is

      detected new type for column 'srid', old type was '4', new type is '23'; refreshing table schema

              Unassigned Unassigned
              robert.coup Robert Coup (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: