Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1184

Can't continue after snapshot is done

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 0.9.3.Final
    • 0.9.2.Final
    • mysql-connector
    • None

    Description

      On MySQL connector, we are testing on production using a slave and a single table that has 71 million records. After the initial snapshot is done (it took about 13 hours) we got below error and didn't run with restart also:

      Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,228] INFO Kafka version : 2.1.0 (org.apache.kafka.common.utils.AppInfoParser:109)
Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,228] INFO Kafka commitId : 809be928f1ae004e (org.apache.kafka.common.utils.AppInfoParser:110)
Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,231] INFO Cluster ID: CdnNlOo2QfC5Jf7wsh25Kg (org.apache.kafka.clients.Metadata:285)
Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,262] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Discovered group coordinator 10.63.6.22:9092 (id: 2147483641 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:654)
Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,270] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Revoking previously assigned partitions [] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:458)
Mar 15 08:10:04 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:04,270] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:486)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,283] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Successfully joined group with generation 1 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:450)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,283] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Setting newly assigned partitions [dbhistory.toyblast-0] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:289)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,286] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Resetting offset for partition dbhistory.toyblast-0 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher:583)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,316] INFO [Consumer clientId=toyblast-connector-dbhistory, groupId=toyblast-connector-dbhistory] Sending LeaveGroup request to coordinator 10.63.6.22:9092 (id: 2147483641 rack: null) (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:782)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,331] INFO MySQL current GTID set 024555da-0127-11e7-adec-0e730df2aa1c:1-143937933,0523b21c-e8ad-11e7-9819-0ea79dd567ba:1-984852811,05929029-af62-11e7-a38c-0ea0544f8398:1-4597232449,0873e912-23b8-11e9-a585-0e37e51d2c30:1-3131944204,0af2831f-3203-11e7-ac5c-0ebe25c21d10:1-1144675264,10a26f94-2782-11e8-83d5-0eba6b56d02a:1-11372299719,14630813-fde0-11e8-aff3-0ecaf9bde85c:1-6800597967,15eb5f8f-ee5e-11e8-a6fb-0ef70282fe3a:1-1814337866,180e86ac-e27f-11e6-9fd2-0e569f91278e:1-130940923,1bfcd1e2-6d57-11e8-95bc-0e0206fb7ab0:1-27164136073,1fdd1c4f-dec3-11e7-91ac-0ede3d301700:1-432943013,1fdec14a-36d3-11e7-9cda-0edc1a04ae22:1-773735874,23a20ec4-5329-11e7-bfe1-0ecfc08fb450:1-1874959742,2589aa94-59a6-11e8-bdcd-0e9d73e2cfa2:1-2159030937,278ca650-081b-11e7-af99-0ea98cf050d0:1-454887951,355291f6-3fc8-11e7-b729-0e78be2a7142:1-2,3df9ad4d-e27f-11e6-b72f-0e21c549c770:1-2428838226,43394f8b-5690-11e7-893f-0e87032ff2fc:1-6217212109,53d4f55e-5419-11e7-9729-0e69018f855a:1-108,5bcc4306-eda6-11e8-a7a0-0e4471c64ebe:1-1061030562,6bdf0bb8-f210-11e7-bb81-0e6b8da43e2c:1-59,6c098bb2-5419-11e7-97d2-0eb5ee13828e:1-17574,71fc13ba-3c06-11e9-ae4a-0ed09c8b803c:1-1684382671,74a85a8c-fc50-11e7-8217-0e2696d2ecd2:1-1,7753ba76-0fac-11e7-b23e-0e652d1ea016:1-1691541305,778601c4-0820-11e7-aedb-0ef36e708f3a:1-713159486,78b1ed79-bcfc-11e8-90ad-0e7c8e7e9a1c:1-379062667,92ca2e6c-ddb1-11e8-869b-0e160d195266:1-458686211,999fc000-0e73-11e7-9275-0efabee2da2e:1-288549915,9ac3793b-2b65-11e8-b907-0ef448f65584:1-3941380971,9efa5ee9-f1fe-11e7-bcc0-0e0710bcb776:1-3957205,a755ac1e-af9f-11e7-96ca-0ed75c87d5d2:1-1208046868,ad50c508-0e73-11e7-b7cd-0e75020f9e04:1-56549808,b3e336a6-518a-11e7-9ea7-0ee6abc7e366:1-5,bc31dcab-e1ac-11e6-ac98-0e2b0f61d5f8:1-2668552160,c1d13d38-0c3d-11e7-b219-0efb2ef4529e:1-101668044,c1efd5ba-ebd3-11e7-ad48-0edb4bc5835e:1-17230343069,c8c67bc3-2b88-11e8-99af-0ed111bea95a:1-4,cd6efae2-fe41-11e8-a246-0ebaa7a67dc8:1-1371604467,ce533dea-10bc-11e7-bc8f-0eb7c95afeaa:1-3877145391,ce5f65bc-1e52-11e9-b81e-0ebd35ba2c86:1-849820969,cf9c33ee-340f-11e8-bb57-0ea1f16ec06e:1-37,d99f37c5-a792-11e8-80d0-0e53e2e53e00:1-474741,eba237f4-0fc2-11e7-b656-0e777952f7fa:1-193700284,f43e819f-34f0-11e7-9189-0e328af992ae:1-2289151198,fc904730-ee5f-11e8-91df-0e8b77bb63ba:1-1345199730,fd7619c3-2564-11e9-ac92-0e6ca5996b5e:1-1847486454 does contain the GTID set required by the connector 024555da-0127-11e7-adec-0e730df2aa1c:1-143937933,0523b21c-e8ad-11e7-9819-0ea79dd567ba:1-984852811,05929029-af62-11e7-a38c-0ea0544f8398:1-4597232449,0873e912-23b8-11e9-a585-0e37e51d2c30:1-3131944204,0af2831f-3203-11e7-ac5c-0ebe25c21d10:1-1144675264,10a26f94-2782-11e8-83d5-0eba6b56d02a:1-11372299719,14630813-fde0-11e8-aff3-0ecaf9bde85c:1-6800597967,15eb5f8f-ee5e-11e8-a6fb-0ef70282fe3a:1-1814337866,180e86ac-e27f-11e6-9fd2-0e569f91278e:1-130940923,1bfcd1e2-6d57-11e8-95bc-0e0206fb7ab0:1-27164136073,1fdd1c4f-dec3-11e7-91ac-0ede3d301700:1-432943013,1fdec14a-36d3-11e7-9cda-0edc1a04ae22:1-773735874,23a20ec4-5329-11e7-bfe1-0ecfc08fb450:1-1874959742,2589aa94-59a6-11e8-bdcd-0e9d73e2cfa2:1-2159030937,278ca650-081b-11e7-af99-0ea98cf050d0:1-454887951,355291f6-3fc8-11e7-b729-0e78be2a7142:1-2,3df9ad4d-e27f-11e6-b72f-0e21c549c770:1-2428838226,43394f8b-5690-11e7-893f-0e87032ff2fc:1-6217212109,53d4f55e-5419-11e7-9729-0e69018f855a:1-108,5bcc4306-eda6-11e8-a7a0-0e4471c64ebe:1-1061030562,6bdf0bb8-f210-11e7-bb81-0e6b8da43e2c:1-59,6c098bb2-5419-11e7-97d2-0eb5ee13828e:1-17574,71fc13ba-3c06-11e9-ae4a-0ed09c8b803c:1-1525187202,74a85a8c-fc50-11e7-8217-0e2696d2ecd2:1-1,7753ba76-0fac-11e7-b23e-0e652d1ea016:1-1691541305,778601c4-0820-11e7-aedb-0ef36e708f3a:1-713159486,78b1ed79-bcfc-11e8-90ad-0e7c8e7e9a1c:1-379062667,92ca2e6c-ddb1-11e8-869b-0e160d195266:1-458686211,999fc000-0e73-11e7-9275-0efabee2da2e:1-288549915,9ac3793b-2b65-11e8-b907-0ef448f65584:1-3941380971,9efa5ee9-f1fe-11e7-bcc0-0e0710bcb776:1-3957205,a755ac1e-af9f-11e7-96ca-0ed75c87d5d2:1-1208046868,ad50c508-0e73-11e7-b7cd-0e75020f9e04:1-56549808,b3e336a6-518a-11e7-9ea7-0ee6abc7e366:1-5,bc31dcab-e1ac-11e6-ac98-0e2b0f61d5f8:1-2668552160,c1d13d38-0c3d-11e7-b219-0efb2ef4529e:1-101668044,c1efd5ba-ebd3-11e7-ad48-0edb4bc5835e:1-17230343069,c8c67bc3-2b88-11e8-99af-0ed111bea95a:1-4,cd6efae2-fe41-11e8-a246-0ebaa7a67dc8:1-1371604467,ce533dea-10bc-11e7-bc8f-0eb7c95afeaa:1-3877145391,ce5f65bc-1e52-11e9-b81e-0ebd35ba2c86:1-849820969,cf9c33ee-340f-11e8-bb57-0ea1f16ec06e:1-37,d99f37c5-a792-11e8-80d0-0e53e2e53e00:1-474741,eba237f4-0fc2-11e7-b656-0e777952f7fa:1-193700284,f43e819f-34f0-11e7-9189-0e328af992ae:1-2289151198,fc904730-ee5f-11e8-91df-0e8b77bb63ba:1-1345199730,fd7619c3-2564-11e9-ac92-0e6ca5996b5e:1-1847486454 (io.debezium.connector.mysql.MySqlConnectorTask:487)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,340] INFO Requested thread factory for connector MySqlConnector, id = toyblast named = binlog-client (io.debezium.util.Threads:247)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,349] INFO Stopping MySQL connector task (io.debezium.connector.mysql.MySqlConnectorTask:428)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,349] INFO WorkerSourceTask{id=toyblast-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask:397)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,349] INFO WorkerSourceTask{id=toyblast-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:414)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,349] ERROR WorkerSourceTask{id=toyblast-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:177)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: org.apache.kafka.connect.errors.ConnectException: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:273)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:47)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:198)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:175)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:219)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.lang.Thread.run(Thread.java:748)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at java.lang.String.substring(String.java:1967)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at com.github.shyiko.mysql.binlog.GtidSet.<init>(GtidSet.java:51)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.GtidSet.<init>(GtidSet.java:42)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.MySqlJdbcContext.purgedGtidSet(MySqlJdbcContext.java:223)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.BinlogReader.doStart(BinlogReader.java:298)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.AbstractReader.start(AbstractReader.java:116)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.ChainedReader.startNextReader(ChainedReader.java:203)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.ChainedReader.start(ChainedReader.java:103)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:256)
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: ... 9 more
Mar 15 08:10:07 bi-dev-cdc1.int.pkgms.net connect-distributed.sh[20972]: [2019-03-15 08:10:07,350] ERROR WorkerSourceTask{id=toyblast-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:178)

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            aykut_jira Aykut Farsak (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: