-
Feature Request
-
Resolution: Duplicate
-
Major
-
None
-
None
-
None
-
False
-
-
False
The following error occurs in the change stream for tables containing the TOKENLIST type. We would like support for the TOKENLIST type.
[2025-09-30 05:34:33,245] ERROR [spannerchangestream|task-0] Task failure, taskUid: spannerchangestream_task-0_afd6b0ac-5654-4781-8da1-bf77354cabb7, java.lang.IllegalArgumentException: Unknown spanner type TOKENLIST at io.debezium.connector.spanner.db.model.schema.Column.parseColumnType(Column.java:93) at io.debezium.connector.spanner.db.model.schema.Column.create(Column.java:52) at io.debezium.connector.spanner.db.model.schema.SpannerSchema$SpannerSchemaBuilder.addColumn(SpannerSchema.java:56) at io.debezium.connector.spanner.db.dao.SchemaDao.getSchema(SchemaDao.java:55) at io.debezium.connector.spanner.db.metadata.SchemaRegistry.forceUpdateSchema(SchemaRegistry.java:153) at io.debezium.connector.spanner.db.metadata.SchemaRegistry.init(SchemaRegistry.java:59) at io.debezium.connector.spanner.task.SynchronizationTaskContext.init(SynchronizationTaskContext.java:194) at io.debezium.connector.spanner.SpannerConnectorTask.start(SpannerConnectorTask.java:229) at io.debezium.connector.spanner.SpannerConnectorTask.start(SpannerConnectorTask.java:59) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:367) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:277) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:478) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:354) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:213) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:268) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) (io.debezium.connector.spanner.SpannerErrorHandler:47)
The table DDL is as follows.
CREATE TABLE TestTable ( id STRING(36) NOT NULL, note STRING(MAX) NOT NULL, note_tokens TOKENLIST AS (TOKENIZE_SUBSTRING(note)) HIDDEN, created TIMESTAMP NOT NULL, updated TIMESTAMP NOT NULL, ) PRIMARY KEY(Id);
Implementing TOKENLIST type in Apache Beam may be a useful reference.
https://github.com/search?q=repo%3Aapache%2Fbeam%20TOKENLIST&type=code
- impacts account
-
DBZ-9526 Support TOKENLIST datatype in Spanner connector
-
- Resolved
-