Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8147

SchemaOnlyRecoverySnapshotter not registered as an SPI service implementation

XMLWordPrintable

    • Moderate

      Here's the related discussion where ccranfor@redhat.com identified the bug from the logs we provided (also copied below):

      https://debezium.zulipchat.com/#narrow/stream/348104-community-mysql-mariadb/topic/issues.20after.20upgrading.20from.202.2E5.20to.202.2E7

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      2.7.1.Final

      What is the connector configuration?

       

      2024-08-13T19:29:36,750pool-2-thread-165INFOi.a.c.i.d.i.DebeziumRecordPublisher(start$lambda$1):63 - Unable to initialize and start connector's task class 'io.debezium.connector.mysql.MySqlConnectorTask' with config: {snapshot.locking.mode=none, connector.class=io.debezium.connector.mysql.MySqlConnector, max.queue.size=8192, include.schema.changes=false, schema.history.internal.store.only.captured.databases.ddl=true, schema.history.internal.file.filename=/tmp/cdc-db-history15112355764930600659/dbhistory.dat, value.converter.replace.null.with.default=false, topic.prefix=db_syozdhyosw, binary.handling.mode=base64, offset.storage.file.filename=/tmp/cdc-state-offset16363759589839396518/offset.dat, decimal.handling.mode=string, converters=boolean, datetime, errors.retry.delay.initial.ms=299, datetime.type=io.airbyte.integrations.source.mysql.cdc.MySQLDateTimeConverter, value.converter=org.apache.kafka.connect.json.JsonConverter, key.converter=org.apache.kafka.connect.json.JsonConverter, database.user=user_syozdhyosw, database.dbname=db_syozdhyosw, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, database.server.id=5727, max.queue.size.in.bytes=268435456, boolean.type=io.airbyte.integrations.source.mysql.cdc.CustomMySQLTinyIntOneToBooleanConverter, errors.retry.delay.max.ms=300, offset.flush.timeout.ms=5000, heartbeat.interval.ms=1000, column.include.list=, database.port=62348, offset.flush.interval.ms=1000, key.converter.schemas.enable=false, database.ssl.mode=DISABLED, schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory, errors.max.retries=0, database.connectionTimeZone=America/Los_Angeles, database.hostname=localhost, database.password={}{}**, value.converter.schemas.enable=false, name=db_syozdhyosw, max.batch.size=2048, table.include.list=, snapshot.mode=schema_only_recovery, database.include.list=db_syozdhyosw}

      What is the captured database version and mode of deployment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      <Your answer>

      What behavior do you expect?

      After upgrading to 2.7.1 from 2.6, we started seeing the following failure: 

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      Suppressed: java.lang.RuntimeException: io.debezium.DebeziumException: Unable to create service io.debezium.snapshot.SnapshotterService
        at io.airbyte.cdk.integrations.debezium.internals.DebeziumRecordPublisher.close(DebeziumRecordPublisher.kt:124)
        at io.airbyte.integrations.source.mysql.cdc.MySqlDebeziumStateUtil.constructInitialDebeziumState(MySqlDebeziumStateUtil.java:254)
        ... 68 more
    Caused by: io.debezium.DebeziumException: Unable to create service io.debezium.snapshot.SnapshotterService
        at io.debezium.service.DefaultServiceRegistry.createService(DefaultServiceRegistry.java:231)
        at io.debezium.service.DefaultServiceRegistry.initializeService(DefaultServiceRegistry.java:204)
        at io.debezium.service.DefaultServiceRegistry.getService(DefaultServiceRegistry.java:79)
        at io.debezium.service.spi.ServiceRegistry.tryGetService(ServiceRegistry.java:40)
        at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:113)
        at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:248)
        at io.debezium.embedded.EmbeddedEngine.startSourceTask(EmbeddedEngine.java:649)
        at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:452)
        at io.debezium.embedded.ConvertingEngineBuilder$1.run(ConvertingEngineBuilder.java:163)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)
    Caused by: io.debezium.DebeziumException: Unable to find schema_only_recovery snapshotter. Please check your configuration.
        at io.debezium.snapshot.SnapshotterServiceProvider.createService(SnapshotterServiceProvider.java:54)
        at io.debezium.snapshot.SnapshotterServiceProvider.createService(SnapshotterServiceProvider.java:30)
        at io.debezium.service.DefaultServiceRegistry.createService(DefaultServiceRegistry.java:115)
        at io.debezium.service.DefaultServiceRegistry.createService(DefaultServiceRegistry.java:224)
        ... 11 more
 
      2024-08-13T19:29:36,665`pool-2-thread-1`65`INFO`o.a.k.c.s.FileOffsetBackingStore(start):63 - Starting FileOffsetBackingStore with file /tmp/cdc-state-offset16363759589839396518/offset.dat
2024-08-13T19:29:36,670`pool-2-thread-1`65`INFO`i.a.c.i.d.i.DebeziumRecordPublisher$start$3(connectorStarted):79 - DebeziumEngine notify: connector started
2024-08-13T19:29:36,675`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(start):242 - Starting MySqlConnectorTask with configuration:
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    connector.class = io.debezium.connector.mysql.MySqlConnector
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    snapshot.locking.mode = none
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    max.queue.size = 8192
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    include.schema.changes = false
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    schema.history.internal.store.only.captured.databases.ddl = true
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    schema.history.internal.file.filename = /tmp/cdc-db-history15112355764930600659/dbhistory.dat
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    value.converter.replace.null.with.default = false
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    topic.prefix = db_syozdhyosw
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    binary.handling.mode = base64
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    offset.storage.file.filename = /tmp/cdc-state-offset16363759589839396518/offset.dat
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    decimal.handling.mode = string
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    converters = boolean, datetime
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    errors.retry.delay.initial.ms = 299
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    datetime.type = io.airbyte.integrations.source.mysql.cdc.MySQLDateTimeConverter
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    value.converter = org.apache.kafka.connect.json.JsonConverter
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    key.converter = org.apache.kafka.connect.json.JsonConverter
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.user = user_syozdhyosw
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.dbname = db_syozdhyosw
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    offset.storage = org.apache.kafka.connect.storage.FileOffsetBackingStore
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.server.id = 5727
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    max.queue.size.in.bytes = 268435456
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    boolean.type = io.airbyte.integrations.source.mysql.cdc.CustomMySQLTinyIntOneToBooleanConverter
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    errors.retry.delay.max.ms = 300
2024-08-13T19:29:36,676`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    offset.flush.timeout.ms = 5000
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    heartbeat.interval.ms = 1000
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    column.include.list =
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.port = 62348
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    offset.flush.interval.ms = 1000
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    key.converter.schemas.enable = false
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.ssl.mode = DISABLED
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    schema.history.internal = io.debezium.storage.file.history.FileSchemaHistory
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    errors.max.retries = 0
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.connectionTimeZone = America/Los_Angeles
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.hostname = localhost
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.password = ********
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    value.converter.schemas.enable = false
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    name = db_syozdhyosw
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    max.batch.size = 2048
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    table.include.list =
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    snapshot.mode = schema_only_recovery
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(lambda$start$0):244 -    database.include.list = db_syozdhyosw
2024-08-13T19:29:36,677`pool-2-thread-1`65`INFO`i.d.c.CommonConnectorConfig(getSourceInfoStructMaker):1684 - Loading the custom source info struct maker plugin: io.debezium.connector.mysql.MySqlSourceInfoStructMaker
2024-08-13T19:29:36,678`pool-2-thread-1`65`INFO`i.d.c.CommonConnectorConfig(getTopicNamingStrategy):1388 - Loading the custom topic naming strategy plugin: io.debezium.schema.DefaultTopicNamingStrategy
2024-08-13T19:29:36,687`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(getPreviousOffsets):501 - Found previous partition offset BinlogPartition{serverName='db_syozdhyosw'} io.debezium.connector.mysql.MySqlPartition@2056030f: {file=binlog.000002, pos=7549}
2024-08-13T19:29:36,749`pool-2-thread-1`65`INFO`i.d.c.c.BaseSourceTask(stop):406 - Stopping down connector
2024-08-13T19:29:36,749`pool-4-thread-1`70`INFO`i.d.j.JdbcConnection(lambda$doClose$4):983 - Connection gracefully closed
2024-08-13T19:29:36,750`pool-2-thread-1`65`INFO`o.a.k.c.s.FileOffsetBackingStore(stop):71 - Stopped FileOffsetBackingStore
2024-08-13T19:29:36,750`pool-2-thread-1`65`INFO`i.a.c.i.d.i.DebeziumRecordPublisher$start$3(connectorStopped):83 - DebeziumEngine notify: connector stopped
2024-08-13T19:29:36,750`pool-2-thread-1`65`INFO`i.a.c.i.d.i.DebeziumRecordPublisher(start$lambda$1):60 - Debezium engine shutdown. Engine terminated successfully : false
2024-08-13T19:29:36,750`pool-2-thread-1`65`INFO`i.a.c.i.d.i.DebeziumRecordPublisher(start$lambda$1):63 - Unable to initialize and start connector's task class 'io.debezium.connector.mysql.MySqlConnectorTask' with config: {snapshot.locking.mode=none, connector.class=io.debezium.connector.mysql.MySqlConnector, max.queue.size=8192, include.schema.changes=false, schema.history.internal.store.only.captured.databases.ddl=true, schema.history.internal.file.filename=/tmp/cdc-db-history15112355764930600659/dbhistory.dat, value.converter.replace.null.with.default=false, topic.prefix=db_syozdhyosw, binary.handling.mode=base64, offset.storage.file.filename=/tmp/cdc-state-offset16363759589839396518/offset.dat, decimal.handling.mode=string, converters=boolean, datetime, errors.retry.delay.initial.ms=299, datetime.type=io.airbyte.integrations.source.mysql.cdc.MySQLDateTimeConverter, value.converter=org.apache.kafka.connect.json.JsonConverter, key.converter=org.apache.kafka.connect.json.JsonConverter, database.user=user_syozdhyosw, database.dbname=db_syozdhyosw, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, database.server.id=5727, max.queue.size.in.bytes=268435456, boolean.type=io.airbyte.integrations.source.mysql.cdc.CustomMySQLTinyIntOneToBooleanConverter, errors.retry.delay.max.ms=300, offset.flush.timeout.ms=5000, heartbeat.interval.ms=1000, column.include.list=, database.port=62348, offset.flush.interval.ms=1000, key.converter.schemas.enable=false, database.ssl.mode=DISABLED, schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory, errors.max.retries=0, database.connectionTimeZone=America/Los_Angeles, database.hostname=localhost, database.password=********, value.converter.schemas.enable=false, name=db_syozdhyosw, max.batch.size=2048, table.include.list=, snapshot.mode=schema_only_recovery, database.include.list=db_syozdhyosw}

            rh-ee-mvitale Mario Fiore Vitale
            yueliab Yue Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: