Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-9515

Supports TOKENLIST type with Spanner connector

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Duplicate
    • Icon: Major Major
    • None
    • None
    • spanner-connector
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      The following error occurs in the change stream for tables containing the TOKENLIST type. We would like support for the TOKENLIST type.

      [2025-09-30 05:34:33,245] ERROR [spannerchangestream|task-0] Task failure, taskUid: spannerchangestream_task-0_afd6b0ac-5654-4781-8da1-bf77354cabb7, java.lang.IllegalArgumentException: Unknown spanner type TOKENLIST
          at io.debezium.connector.spanner.db.model.schema.Column.parseColumnType(Column.java:93)
          at io.debezium.connector.spanner.db.model.schema.Column.create(Column.java:52)
          at io.debezium.connector.spanner.db.model.schema.SpannerSchema$SpannerSchemaBuilder.addColumn(SpannerSchema.java:56)
          at io.debezium.connector.spanner.db.dao.SchemaDao.getSchema(SchemaDao.java:55)
          at io.debezium.connector.spanner.db.metadata.SchemaRegistry.forceUpdateSchema(SchemaRegistry.java:153)
          at io.debezium.connector.spanner.db.metadata.SchemaRegistry.init(SchemaRegistry.java:59)
          at io.debezium.connector.spanner.task.SynchronizationTaskContext.init(SynchronizationTaskContext.java:194)
          at io.debezium.connector.spanner.SpannerConnectorTask.start(SpannerConnectorTask.java:229)
          at io.debezium.connector.spanner.SpannerConnectorTask.start(SpannerConnectorTask.java:59)
          at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:367)
          at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:277)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:478)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:354)
          at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:213)
          at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:268)
          at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:78)
          at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
       (io.debezium.connector.spanner.SpannerErrorHandler:47)
       

      The table DDL is as follows.

      CREATE TABLE TestTable (
        id STRING(36) NOT NULL,
        note STRING(MAX) NOT NULL,
        note_tokens TOKENLIST AS (TOKENIZE_SUBSTRING(note)) HIDDEN,
        created TIMESTAMP NOT NULL,
        updated TIMESTAMP NOT NULL,
      ) PRIMARY KEY(Id);

      Implementing TOKENLIST type in Apache Beam may be a useful reference.

      https://github.com/search?q=repo%3Aapache%2Fbeam%20TOKENLIST&type=code

              Unassigned Unassigned
              laughingman7743 Tomoyuki Nakamura
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: