Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7136

Failure during snapshot initial of Google Cloud SQL(MySQL)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • None
    • debezium-server
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      • I'm trying to upgrade from version 1.9.7 but the offset state cannot be used in version 2.x

      https://issues.redhat.com/browse/DBZ-6789

      • Snapshot was started again, this error happens only when the database has a lot of records.
      • debezium sends data pub/sub, and after pub/sub sends to Cloud Storage. 

      What Debezium connector do you use and what version?

      Mysql connector 2.3.2, 2.4.0

      What is the connector configuration?

      debezium.sink.type=pubsub    debezium.source.offset.storage.file.filename=offsetsNew.dat    debezium.source.schema.history.internal=io.debezium.storage.file.history.FileSchemaHistory    debezium.source.schema.history.internal.file.filename=db-history.dat    debezium.source.schema.history.internal.skip.unparseable.ddl=true    debezium.transforms=singleTopic    debezium.transforms.singleTopic.type=io.debezium.transforms.ByLogicalTableRouter    debezium.transforms.singleTopic.topic.regex=([^.]+).*    debezium.transforms.singleTopic.topic.replacement=$1    debezium.sink.pubsub.project.id    debezium.source.database.server.id=199999    debezium.source.topic.prefix    debezium.source.connector.class    debezium.source.database.hostname    debezium.source.database.port    debezium.source.database.user    debezium.source.database.password    debezium.source.database.server.name    debezium.source.database.include.list 

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      Google Cloud SQL MySQL 8.0

      Kubernetes cluster statefulset, GKE

      What behaviour do you expect?

      Snapshot will be made successfully.

      What behaviour do you see?

      Snapshot was not completed successfully, it will be re-executed upon connector restart

      RelationalSnapshotChangeEventSource","level":"ERROR","message":"Error during snapshot","threadName":"debezium-mysqlconnector-debezium_mysql_cards_test-change-event-source-coordinator","threadId":30,"mdc":{"dbz.taskId":"0","dbz.connectorName":"debezium_mysql_cards_test","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1,"exception":{"refId":1,"exceptionType":"java.lang.InterruptedException","message":null,"frames":[{"class<span class="code-quote">":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","method":"reportInterruptAfterWait","line":2056},{"class<span class="code-quote">":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","method":"await","line":2090},{"class<span class="code-quote">":"java.util.concurrent.LinkedBlockingQueue","method":"take","line":433},{"class<span class="code-quote">":"java.util.concurrent.ExecutorCompletionService","method":"take","line":200},{"class<span class="code-quote">":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEvents","line":463},{"class<span class="code-quote">":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"doExecute","line":165},{"class<span class="code-quote">":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","method":"execute","line":92},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":253},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":237},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"executeChangeEventSources","line":189},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"lambda$start$0","line":137},{"class<span class="code-quote">":"java.util.concurrent.Executors$RunnableAdapter","method":"call","line":515},{"class<span class="code-quote">":"java.util.concurrent.FutureTask","method":"run","line":264},{"class<span class="code-quote">":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1128},{"class<span class="code-quote">":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":628},{"class<span class="code-quote">":"java.lang.Thread","method":"run","line":829}]}}{"timestamp":"2023-11-13T13:20:29.715Z","sequence":401,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","level":"WARN","message":"Snapshot was interrupted before completion","threadName":"debezium-mysqlconnector-debezium_mysql_cards_test-change-event-source-coordinator","threadId":30,"mdc":{"dbz.taskId":"0","dbz.connectorName":"debezium_mysql_cards_test","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.715Z","sequence":402,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","level":"INFO","message":"Snapshot - Final stage","threadName":"debezium-mysqlconnector-debezium_mysql_cards_test-change-event-source-coordinator","threadId":30,"mdc":{"dbz.taskId":"0","dbz.connectorName":"debezium_mysql_cards_test","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.715Z","sequence":403,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","level":"WARN","message":"Snapshot was not completed successfully, it will be re-executed upon connector restart","threadName":"debezium-mysqlconnector-debezium_mysql_cards_test-change-event-source-coordinator","threadId":30,"mdc":{"dbz.taskId":"0","dbz.connectorName":"debezium_mysql_cards_test","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.716Z","sequence":404,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.ChangeEventSourceCoordinator","level":"WARN","message":"Change event source executor was interrupted","threadName":"debezium-mysqlconnector-debezium_mysql_cards_test-change-event-source-coordinator","threadId":30,"mdc":{"dbz.taskId":"0","dbz.connectorName":"debezium_mysql_cards_test","dbz.connectorType":"MySQL","dbz.connectorContext":"snapshot"},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1,"exception":{"refId":1,"exceptionType":"java.lang.InterruptedException","message":null,"frames":[{"class<span class="code-quote">":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","method":"reportInterruptAfterWait","line":2056},{"class<span class="code-quote">":"java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject","method":"await","line":2090},{"class<span class="code-quote">":"java.util.concurrent.LinkedBlockingQueue","method":"take","line":433},{"class<span class="code-quote">":"java.util.concurrent.ExecutorCompletionService","method":"take","line":200},{"class<span class="code-quote">":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"createDataEvents","line":463},{"class<span class="code-quote">":"io.debezium.relational.RelationalSnapshotChangeEventSource","method":"doExecute","line":165},{"class<span class="code-quote">":"io.debezium.pipeline.source.AbstractSnapshotChangeEventSource","method":"execute","line":92},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":253},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"doSnapshot","line":237},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"executeChangeEventSources","line":189},{"class<span class="code-quote">":"io.debezium.pipeline.ChangeEventSourceCoordinator","method":"lambda$start$0","line":137},{"class<span class="code-quote">":"java.util.concurrent.Executors$RunnableAdapter","method":"call","line":515},{"class<span class="code-quote">":"java.util.concurrent.FutureTask","method":"run","line":264},{"class<span class="code-quote">":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","line":1128},{"class<span class="code-quote">":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","line":628},{"class<span class="code-quote">":"java.lang.Thread","method":"run","line":829}]}}{"timestamp":"2023-11-13T13:20:29.722Z","sequence":405,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.pipeline.signal.SignalProcessor","level":"INFO","message":"SignalProcessor stopped","threadName":"pool-7-thread-1","threadId":22,"mdc":{},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.729Z","sequence":406,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.jdbc.JdbcConnection","level":"INFO","message":"Connection gracefully closed","threadName":"pool-14-thread-1","threadId":50,"mdc":{},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.734Z","sequence":407,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"org.apache.kafka.connect.storage.FileOffsetBackingStore","level":"INFO","message":"Stopped FileOffsetBackingStore","threadName":"pool-7-thread-1","threadId":22,"mdc":{},"ndc":"","hostName":"debezium-mysql-cards-test-0","processName":"io.debezium.server.Main","processId":1}{"timestamp":"2023-11-13T13:20:29.737Z","sequence":408,"loggerClassName":"org.slf4j.impl.Slf4jLogger","loggerName":"io.debezium.server.ConnectorLifecycle","level":"ERROR","message":"Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: java.util.concurrent.TimeoutException: Waited 30000 milliseconds (plus 258941 nanoseconds delay) for ListFuture@11eb7b58[status=PENDING, info=[futures=[com.google.api.core.AbstractApiFuture$InternalSettableFuture@46a5bb3f[status=SUCCESS, result=[java.lang.String@78cc03e0]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@42df5d1f[status=SUCCESS, result=[java.lang.String@8a4cfa7]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@6333e1cf[status=SUCCESS, result=[java.lang.String@7b783d4a]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@4377b09f[status=SUCCESS, result=[java.lang.String@6d0086a1]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@11c12f57[status=SUCCESS, result=[java.lang.String@2b69c248]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@55e46cf2[status=SUCCESS, result=[java.lang.String@3cb8cd61]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@249bf636[status=SUCCESS, result=[java.lang.String@41050b6f]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@37706c41[status=SUCCESS, result=[java.lang.String@4dd97c0b]], com.google.api.core.AbstractApiFuture$InternalSettableFuture@360819c0[status=SUCCESS, result=[java.lang.String@5fbf221d]], com.google.api.core. 

      Do you see the same behaviour using the latest relesead Debezium version?

      For 2.4.0, the same behaviour 

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

      logs-test

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      <Your answer>

              Unassigned Unassigned
              valynkoandrei Andrey Valynko (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: