We recently upgraded debezium from 2.2.1.Final to 2.4.0.Alpha1

During off hours we pause some streaming mysql connectors and create snapshot only connectors that are responsible for performing a data snapshot on a specific set of tables that have had relevant DDL changes that the binlog does not pick up.

We Pause the streaming connectors...

We create Snapshot Only connectors with the same config and then a table include list and snapshot.mode set to "Initial Only"

configs added to Snapshot only connector

snapshot.mode    "initial_only"

snapshot.max.threads    "4"

snapshot.tables.order.by.row.count    "descending"

notification.sink.topic.name    "snapshot_only_connector_notifications"

notification.enabled.channels    "sink"

In the past this worked without issue but after switching to using Debezium 2.4.0.Alpha1 we started seeing this error randomly occur in the snapshot connectors...

 
org.apache.kafka.connect.errors.ConnectException: Source offset 'file' parameter is missing at io.debezium.connector.mysql.MySqlOffsetContext$Loader.load(MySqlOffsetContext.java:176) at io.debezium.connector.mysql.MySqlOffsetContext$Loader.load(MySqlOffsetContext.java:161) at io.debezium.connector.common.OffsetReader.lambda$offsets$0(OffsetReader.java:50) at java.base/java.util.Collections$SingletonSet.forEach(Collections.java:4797) at io.debezium.connector.common.OffsetReader.offsets(OffsetReader.java:45) at io.debezium.connector.common.BaseSourceTask.getPreviousOffsets(BaseSourceTask.java:378) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:92) at io.debezium.connector.common.BaseSourceTask.startIfNeededAndPossible(BaseSourceTask.java:260) at io.debezium.connector.common.BaseSourceTask.poll(BaseSourceTask.java:169) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.poll(AbstractWorkerSourceTask.java:462) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:351) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:204) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259) at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75) at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
 
We cant use any other version of debezium due to lag in snapshots being introduced and other bugs we found, and this is also the only version that works for snapshot notifications that does not have the mentioned bugs.
 
Curious if this error has any significance or meaning that we can debug. Why is the connector looking for a previous offset? This is an initial only snapshot