Bug report

For bug reports, provide this information, please:

What Debezium connector do you use and what version?

Spanner-Connector v2.2.0.Alpha1

What is the connector configuration?

{"name": "livestream-connector","config":

{ "connector.class": "io.debezium.connector.spanner.SpannerConnector", "tasks.max": "1", "gcp.spanner.change.stream": "live-table", "gcp.spanner.project.id": "project-stag, "gcp.spanner.instance.id": "development-instance", "gcp.spanner.database.id": "live-db", "gcp.spanner.credentials.path": "/app/services-sa.json", "key.converter.schemas.enable":false, "value.converter.schemas.enable":false }

}

What is the captured database version and mode of depoyment?

GCP Managed Spanner DB and Kafka connect deployed on K8s.

What behaviour do you expect?

The task should be able to start without error.

What behaviour do you see?

Illegal state exception when starting task.

Do you see the same behaviour using the latest relesead Debezium version?

Yes.

Do you have the connector logs, ideally from start till finish?

Yes.
[2023-03-23 08:12:03,998] INFO Stopping task cdc-platform-livestream_member_prod_4_task-0_d0a6306b-f92f-4290-9480-ad2003abefef, spannerMeter (io.debezium.connector.spanner.SpannerConnectorTask)
[2023-03-23 08:12:04,024] INFO Task cdc-platform-livestream_member_prod_4_task-0_d0a6306b-f92f-4290-9480-ad2003abefef was stopped (io.debezium.connector.spanner.SpannerConnectorTask)
[2023-03-23 08:12:04,024] WARN WorkerSourceTask{id=cdc-platform-livestream_member_prod_4-0} failed to poll records from SourceTask. Will retry operation. (org.apache.kafka.connect.runtime.AbstractWorkerSourceTask)
org.apache.kafka.connect.errors.RetriableException: An exception occurred in the change event producer. This connector will be restarted.
        at io.debezium.connector.spanner.SpannerErrorHandler.setProducerThrowable(SpannerErrorHandler.java:57)
        at io.debezium.connector.spanner.SpannerStreamingChangeEventSource.processFailure(SpannerStreamingChangeEventSource.java:272)
        at io.debezium.connector.spanner.SpannerStreamingChangeEventSource.lambda$startProcessing$1(SpannerStreamingChangeEventSource.java:208)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: livestream_member.livestream_id primary key data is incorrect in registry
        at io.debezium.connector.spanner.db.metadata.SchemaValidator.validate(SchemaValidator.java:46)
        at io.debezium.connector.spanner.db.metadata.SchemaRegistry.validate(SchemaRegistry.java:89)
        at io.debezium.connector.spanner.db.metadata.SchemaRegistry.checkSchema(SchemaRegistry.java:68)
        at io.debezium.connector.spanner.SpannerStreamingChangeEventSource.processDataChangeEvent(SpannerStreamingChangeEventSource.java:219)
        at io.debezium.connector.spanner.SpannerStreamingChangeEventSource.lambda$startProcessing$1(SpannerStreamingChangeEventSource.java:176)
        ... 1 more
[2023-03-23 08:12:04,032] INFO Awaiting end of restart backoff period after a retriable error (io.debezium.connector.common.BaseSourceTask)How to reproduce the issue using our tutorial deployment?

Create a secondary index on primary column of table to reproduce this.

Implementation ideas (optional)

Have made join at application level rather than at db level.