Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
None
-
False
Description
Bug report
I am using SqlServerConnector in snapshot mode (INITIAL_ONLY) for fetching records from a table that has space in the name eg ORDER DETAILS. I am trying to fetch limited no of records from the table and so I have overridden the select statement using property:
snapshot.select.statement.overrides.dbo.order details
The override works for tables without space in name, but throws exception for the above config. Ref: https://debezium.io/documentation/reference/stable/connectors/sqlserver.html#sqlserver-property-snapshot-select-statement-overrides
What Debezium connector do you use and what version?
The issue seem to exist on debezium-core-1.9.2.Final
What is the connector configuration?
connector.class = io.debezium.connector.sqlserver.SqlServerConnector
snapshot.select.statement.overrides.dbo.Order Details = SELECT TOP 50 * FROM [dbo].[Order Details]
include.schema.changes = false
value.converter = org.apache.kafka.connect.json.JsonConverter
key.converter = org.apache.kafka.connect.json.JsonConverter
database.user = test
database.dbname = test
offset.storage = org.apache.kafka.connect.storage.MemoryOffsetBackingStore
snapshot.select.statement.overrides = dbo.Order Details
time.precision.mode = connect
database.server.name = test
offset.flush.timeout.ms = 5000
event.processing.failure.handling.mode = fail
offset.flush.interval.ms = 60000
internal.key.converter = org.apache.kafka.connect.json.JsonConverter
database.hostname = localhost
database.password = ***
name = sql_server_connector
internal.value.converter = org.apache.kafka.connect.json.JsonConverter
snapshot.mode = initial_only
database.history = io.debezium.relational.history.MemoryDatabaseHistory
What is the captured database version and mode of depoyment?
I am using debezium embedded engine in my application (debezium-embedded-1.9.2.Final)
What behaviour do you expect?
The same table name (ORDER DETAILS with space) with below configuration works. fine. Similarly, it should work for overridden select statement as well.
table.include.list = dbo.Order Details
What behaviour do you see?
The connector stops with an exception encountered while parsing tableId in TableIdParser.java class.
As what I understand debugging this issue, if config has select override, the below method id called to get the overrides by Table:
RelationalDatabaseConnectorConfig.getSnapshotSelectOverridesByTable()
In the above method, for each table, the table name (schemaname.tablename) is tokenised and parsed to extract the schema name and table name in list of parts eg. ["dbo", "ORDER DETAILS"]
TableIdParser.parse(table)
While parsing the second part (after DOT separator), it iterate over the table name char array in IN_IDENTIFIER -> handleCharacter(...) and identifies whitespace as indication for BEFORE_SEPARATOR. In the next iteration, char 'D' is encountered and ParsingState being BEFORE_SEPARATOR, it falls into exception block, resulting in below exception:
Unexpected input: 'D'
Do you have the connector logs, ideally from start till finish?
Stacktrace:
org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped. at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: io.debezium.DebeziumException: java.lang.IllegalArgumentException: Unexpected input: D at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85) at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155) at io.debezium.connector.sqlserver.SqlServerChangeEventSourceCoordinator.executeChangeEventSources(SqlServerChangeEventSourceCoordinator.java:71) at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) ... 6 common frames omitted Caused by: java.lang.IllegalArgumentException: Unexpected input: D at io.debezium.relational.TableIdParser$ParsingState$3.handleCharacter(TableIdParser.java:144) at io.debezium.relational.TableIdParser$TableIdTokenizer.tokenize(TableIdParser.java:64) at io.debezium.text.TokenStream.start(TokenStream.java:446) at io.debezium.relational.TableIdParser.parse(TableIdParser.java:31) at io.debezium.relational.TableId.parseParts(TableId.java:51) at io.debezium.relational.TableId.parse(TableId.java:40) at io.debezium.relational.TableId.parse(TableId.java:27) at io.debezium.relational.RelationalDatabaseConnectorConfig.getSnapshotSelectOverridesByTable(RelationalDatabaseConnectorConfig.java:804) at io.debezium.relational.RelationalSnapshotChangeEventSource.determineSnapshotSelect(RelationalSnapshotChangeEventSource.java:433) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEventsForTable(RelationalSnapshotChangeEventSource.java:340) at io.debezium.relational.RelationalSnapshotChangeEventSource.createDataEvents(RelationalSnapshotChangeEventSource.java:313) at io.debezium.relational.RelationalSnapshotChangeEventSource.doExecute(RelationalSnapshotChangeEventSource.java:129) at io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76) ... 9 common frames omitted
Implementation ideas (optional)
Could we use the occurrence of [] in the below property as an indication of start and end of schema or table name similar to what we have in query ? ex
snapshot.select.statement.overrides = [dbo].[Order Details]
