Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5198

Debezuim connector fails at parsing select statement overrides when table name has space

    XMLWordPrintable

Details

    Description

      Bug report

      I am using SqlServerConnector in snapshot mode (INITIAL_ONLY) for fetching records from a table that has space in the name eg ORDER DETAILS. I am trying to fetch limited no of records from the table and so I have overridden the select statement using property:

      snapshot.select.statement.overrides.dbo.order details

      The override works for tables without space in name, but throws exception for the above config. Ref: https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-property-snapshot-select-statement-overrides

      What Debezium connector do you use and what version?

      The issue seem to exist on debezium-core-1.9.2.Final

      What is the connector configuration?

      connector.class = io.debezium.connector.sqlserver.SqlServerConnector
      snapshot.select.statement.overrides.dbo.Order Details = SELECT TOP 50 * FROM [dbo].[Order Details]
      include.schema.changes = false
      value.converter = org.apache.kafka.connect.json.JsonConverter
      key.converter = org.apache.kafka.connect.json.JsonConverter
      database.user = test
      database.dbname = test
      offset.storage = org.apache.kafka.connect.storage.MemoryOffsetBackingStore
      snapshot.select.statement.overrides = dbo.Order Details
      time.precision.mode = connect
      database.server.name = test
      offset.flush.timeout.ms = 5000
      event.processing.failure.handling.mode = fail
      offset.flush.interval.ms = 60000
      internal.key.converter = org.apache.kafka.connect.json.JsonConverter
      database.hostname = localhost
      database.password = ***
      name = sql_server_connector
      internal.value.converter = org.apache.kafka.connect.json.JsonConverter
      snapshot.mode = initial_only
      database.history = io.debezium.relational.history.MemoryDatabaseHistory
      

      What is the captured database version and mode of depoyment?

      I am using debezium embedded engine in my application (debezium-embedded-1.9.2.Final)

      What behaviour do you expect?

      The same table name (ORDER DETAILS with space) with below configuration works. fine. Similarly, it should work for overridden select statement as well.

      table.include.list = dbo.Order Details

      What behaviour do you see?

      The connector stops with an exception encountered while parsing tableId in TableIdParser.java class.

      As what I understand debugging this issue, if config has select override, the below method id called to get the overrides by Table:

      RelationalDatabaseConnectorConfig.getSnapshotSelectOverridesByTable()

      In the above method, for each table, the table name (schemaname.tablename) is tokenised and parsed to extract the schema name and table name in list of parts eg. ["dbo", "ORDER DETAILS"]

      TableIdParser.parse(table)
      

      While parsing the second part (after DOT separator), it iterate over the table name char array in IN_IDENTIFIER -> handleCharacter(...) and identifies whitespace as indication for BEFORE_SEPARATOR. In the next iteration, char 'D' is encountered and ParsingState being  BEFORE_SEPARATOR, it falls into exception block, resulting in below exception:

      Unexpected input: 'D'

      Do you have the connector logs, ideally from start till finish?

      Stacktrace:

      org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
          at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: io.debezium.DebeziumException: java.lang.IllegalArgumentException: Unexpected input: D
          at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155)
          at io.debezium.connector.sqlserver.SqlServerChangeEventSourceCoordinator.executeChangeEventSources(SqlServerChangeEventSourceCoordinator.java:71)
          at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109)
          ... 6 common frames omitted
      Caused by: java.lang.IllegalArgumentException: Unexpected input: D
          at io.debezium.relational.TableIdParser$ParsingState$3.handleCharacter(TableIdParser.java:144)
          at io.debezium.relational.TableIdParser$TableIdTokenizer.tokenize(TableIdParser.java:64)
          at io.debezium.text.TokenStream.start(TokenStream.java:446)
          at io.debezium.relational.TableIdParser.parse(TableIdParser.java:31)
          at io.debezium.relational.TableId.parseParts(TableId.java:51)
          at io.debezium.relational.TableId.parse(TableId.java:40)
          at io.debezium.relational.TableId.parse(TableId.java:27)
          at io.debezium.relational.RelationalDatabaseConnectorConfig.getSnapshotSelectOverridesByTable(RelationalDatabaseConnectorConfig.java:804)
          at io.debezium.relational.RelationalSnapshotChangeEventSource.determineSnapshotSelect(RelationalSnapshotChangeEventSource.java:433)
          at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:340)
          at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:313)
          at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:129)
          at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
          ... 9 common frames omitted

      Implementation ideas (optional)

      Could we use the occurrence of [] in the below property as an indication of start and end of schema or table name similar to what we have in query ? ex 

      snapshot.select.statement.overrides = [dbo].[Order Details]

      Attachments

        Activity

          People

            vjuranek@redhat.com Vojtech Juranek
            poonam.meghnani Poonam Meghnani
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: