My team is trying to upgrade to Debezium 2.6.1.Final from 2.2.1.Final. We currently run connectors for both mySQL 8 and mySQL 5 databases. When testing 2.6.1.Final in our staging env we have found that an initial data snapshot takes much longer than normal with the same configs as 2.2.1.Final. Specifically the Data Snapshot , where Debezium uses a transaction and REPEATABLE_READ to generate the read records from each table. We are using "when_needed" snapshots and "minimal_percona".

Testing on MySQL 5 host with 700,000+ tables:

Current Prod Setup: Debezium 2.2.1.Final + mysql-connector-j-8.0.31: Fully snapshots 700,000 tables in less than 10 hours

Debezium 2.6.1.Final + mysql-connector-j-8.0.33: Takes over 3 days to snapshot the same MySQL host. (we downgraded the mysql plugin to be compatible)

Debezium 2.6.1.Final + mysql-connector-j-8.0.31: Takes over 3 days to snapshot the same MySQL host. (we downgraded the mysql plugin to match prod exactly)

Debezium 2.6.1.Final + mysql-connector-j-8.3.0: Takes over 3 days to snapshot the same MySQL host.

We ran the same tests on a mysql 8 host and did not see any improvements with different driver versions.

We are tentatively concluding that there must be some change in the Debezium code that is significantly slowing down the Data Snapshot as we have held all other variables constant across versions.

We previously ran into an issue where the schema snapshot was taking quite a while on older versions of debezium, but that was fixed, in this case, the schema snapshots are finishing in nearly the same amount of time, but the data snapshot is significantly different.

We have actually not gotten a snapshot to complete on 2.6.1.Final for a large host because the metadata lock is held for so long on the tables we run into issues with the prod hosts and have to cancel them.

The mysql user has the same permissions as prod.

What is the connector configuration?

These 2 were added as a last effort to speed up and track snapshotting but did not make a significant difference.

What is the captured database version and mode of depoyment?

MySQL rds hosts on version 5 and 8

What behavior do you expect?

We expect the data snapshot to complete in the same amount of time as Debezium 2.2.1.Final.

What behavior do you see?

Data Snapshot taking multiple days, and causing significant lag because the metadata locks are held on the mysql database for so long, and those databases are replicas that start to get blocked and fall behind.

Do you see the same behavior using the latest released Debezium version?

Yes, issue appears on this version.

Do you have the connector logs, ideally from start till finish?

The logs are the same as other versions, its just that the exporting data from table {} logs are running for days.

How to reproduce the issue using our tutorial deployment?

You most likely cant which is the issue, we run debezium at a scale that usually finds issues that havent been tested for.