Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5646

Debezium stops processing table changes and the process doesn't die

    XMLWordPrintable

Details

    • False
    • None
    • False

    Description

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      debezium/server 1.9.5.Final with the following connector/sink type.

      debezium.sink.type=pubsub
      debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector

      What is the connector configuration?

      debezium.sink.pubsub.ordering.enabled=true
      debezium.sink.pubsub.project.id=kenect-service
      debezium.sink.type=pubsub
      debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
      debezium.source.database.dbname=myDatabase
      debezium.source.database.history.file.filename=/debezium/data/history.dat
      debezium.source.database.history.skip.unparseable.ddl=true
      debezium.source.database.history.store.only.captured.tables.ddl=true
      debezium.source.database.history=io.debezium.relational.history.FileDatabaseHistory
      debezium.source.database.hostname=127.0.0.1
      debezium.source.database.port=3306
      debezium.source.database.server.name=myDatabase
      debezium.source.event.processing.failure.handling.mode=warn
      debezium.source.include.schema.changes=false
      debezium.source.inconsistent.schema.handling.mode=warn
      debezium.source.key.converter.schemas.enable=false
      debezium.source.max.queue.size=81290
      debezium.source.offset.flush.interval.ms=0
      debezium.source.offset.storage.file.filename=/debezium/data/offsets.dat
      debezium.source.tombstones.on.delete=false
      debezium.source.snapshot.locking.mode=none
      debezium.source.snapshot.mode=schema_only
      debezium.source.table.include.list=myDatabase.channel,myDatabase.contact_channel,myDatabase.conv,myDatabase.heartbeat,myDatabase.local_destination,myDatabase.message,myDatabase.message_resource,myDatabase.team,myDatabase.team_member,myDatabase.notification_subscription,myDatabase.legal_mat
      debezium.source.value.converter.schemas.enable=false
      debezium.transforms.Reroute.topic.regex=[^.]+(.*)
      debezium.transforms.Reroute.topic.replacement=debezium$1
      debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      Running debezium in a kubernetes pod using the debezium/server image 1.9.0.Final. The database is a Google Cloudsql mysql version 8 database. This issue exists with both mysql5 and mysql8. At least in Google CloudSql.

      What behaviour do you expect?

      Function normally.

      What behaviour do you see?

      Debezium stops processing any changes until we restart the debezium process.

      Do you see the same behaviour using the latest relesead Debezium version?

      This behavior exists in version 1.9.0.Final and also 1.9.5.Final

      Do you have the connector logs, ideally from start till finish?

      Logs provided before and near the incident. Incident started at time period 7:57 and a restart was issued at 8:03

      2022-09-21 03:48:02.473 MDT 1 records sent during previous 02:00:01.592, last recorded offset: {transaction_id=null, ts_sec=1663753682, file=mysql-bin.021685, pos=94112103, gtids=06ea70dd-1f81-11ed-8866-4201ac1fd03a:1-35376948,0c51eaff-74b8-11e8-88c1-42010a800129:1-62523206,6b50bb0b-99fe-11e9-abaf-42010a800591:1-1032604843,8300061e-cb5b-11e8-8754-42010a80013e:1-34594475,ee829197-2638-11e8-b4d8-42010a80036b:1-12687247, row=1, server_id=2627354887, event=3}
      2022-09-21 05:48:01.636 MDT 3 records sent during previous 01:59:59.163, last recorded offset: {transaction_id=null, ts_sec=1663760881, file=mysql-bin.021685, pos=101217646, gtids=06ea70dd-1f81-11ed-8866-4201ac1fd03a:1-35385785,0c51eaff-74b8-11e8-88c1-42010a800129:1-62523206,6b50bb0b-99fe-11e9-abaf-42010a800591:1-1032604843,8300061e-cb5b-11e8-8754-42010a80013e:1-34594475,ee829197-2638-11e8-b4d8-42010a80036b:1-12687247, row=1, server_id=2627354887, event=5}
      2022-09-21 07:48:01.171 MDT 12 records sent during previous 01:59:59.535, last recorded offset: {transaction_id=null, ts_sec=1663768080, file=mysql-bin.021686, pos=79661307, gtids=06ea70dd-1f81-11ed-8866-4201ac1fd03a:1-35473297,0c51eaff-74b8-11e8-88c1-42010a800129:1-62523206,6b50bb0b-99fe-11e9-abaf-42010a800591:1-1032604843,8300061e-cb5b-11e8-8754-42010a80013e:1-34594475,ee829197-2638-11e8-b4d8-42010a80036b:1-12687247, row=1, server_id=2627354887, event=2}
      2022-09-21 08:03:27.014 MDT Stopped reading binlog after 0 events, last recorded offset: {transaction_id=null, ts_sec=1663769006, file=mysql-bin.021686, pos=97118600, gtids=06ea70dd-1f81-11ed-8866-4201ac1fd03a:1-35491395,0c51eaff-74b8-11e8-88c1-42010a800129:1-62523206,6b50bb0b-99fe-11e9-abaf-42010a800591:1-1032604843,8300061e-cb5b-11e8-8754-42010a80013e:1-34594475,ee829197-2638-11e8-b4d8-42010a80036b:1-12687247, server_id=2627354887, event=1}
      2022-09-21 08:03:27.047 MDT Received request to stop the engine
      2022-09-21 08:03:27.047 MDT Stopping the embedded engine
      2022-09-21 08:03:27.047 MDT Waiting for PT5M for connector to stop

      How to reproduce the issue using our tutorial deployment?

      Being able to reproduce on demand has not been identified. The issue happens between a few hours to a few days before presenting.

      This behavior does exist on two separate instances of debezium, connecting to two different mysql 8 databases with similar configuration.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            bgridley03 Ben Gridley (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: