Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5849

Debezium 2.0 MySQL try to update/get info about a table in a schema not listed in the connector and fails.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • 2.0.0.Final
    • None
    • False
    • None
    • False

      In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.

      Bug report

      For bug reports, provide this information, please:

      What Debezium connector do you use and what version?

      we upgrade from 1.8 to 2.0

      What is the connector configuration?

      {     "connector.class": "io.debezium.connector.mysql.MySqlConnector",     "snapshot.locking.mode": "none",     "_comment": "Connector config for zcs-04_qas",     "database.history.kafka.recovery.poll.interval.ms": "86400000",     "drop.tombstones": "false",     "database.history.skip.unparseable.ddl": "true",     "drop.deletes": "false",     "_comment": "MySQL settings",     "database.hostname": "xxxxxx",     "database.port": "xxxxx",     "database.user": "xxxxxx",     "database.password": "xxxxx",     "database.connectionTimeZone": "Europe/Brussels",     "database.server.id": "2329",     "database.server.name": "zcs-04_qas",     "database.include.list": "zcs",     "schema.history.internal.kafka.topic": "zcs-04_qas.history",     "schema.history.internal.kafka.bootstrap.servers": "xxxxxx.hrx.erp:9092",     "database.history.kafka.bootstrap.servers": "xxxxxxx.hrx.erp:9092",     "ddl.parser.mode": "antlr",     "include.schema.changes": "true",     "decimal.handling.mode": "string",     "connect.keep.alive": "true",     "snapshot.new.tables": "parallel",     "topic.prefix": "zcs-04_qas",     "snapshot.mode": "when_needed",     "table.include.list": "zcs.conf_pay_elements,zcs.conf_wage_type_category_pay_elements,zcs.employments,zcs.enterprise_store_communications,zcs.enterprise_store_cost_assignments,zcs.enterprise_store_deployments,zcs.enterprise_store_personal_data,zcs.enterprise_store_profiles,zcs.fcs_countries,zcs.gccs,zcs.lccs,zcs.pay_calendars,zcs.pay_groups,zcs.pay_periods,zcs.wage_type_categories,zcs.wage_type_report_sources,zcs.users,zcs.lccs_users,zcs.glrep_rows_staging2,zcs.glrep_file2",     "_comment": "Kakfa connector settings",     "poll.interval.ms": "5000",     "_comment": "Kafka converter settings",     "value.converter.schema.registry.url": "http://xxxxxxxx.hrx.erp:8081",     "value.converter": "io.confluent.connect.avro.AvroConverter",     "key.converter": "io.confluent.connect.avro.AvroConverter",     "key.converter.schema.registry.url": "http://xxxxxxxxx.hrx.erp:8081",     "internal.key.converter": "org.apache.kafka.connect.json.JsonConverter",     "internal.value.converter": "org.apache.kafka.connect.json.JsonConverter"   }

      What is the captured database version and mode of depoyment?

      (E.g. on-premises, with a specific cloud provider, etc.)

      MySQL 7.x

      What behaviour do you expect?

      The table and schema no listed/set in the connector configuration are IGNORED.
      When a database as a lot of schemas it generates a lot of unnecessary data collect, logs.

      Based on the configuration of the connector,

      • only the schema/db specified should be taken into account
      • only the tables list in the table list parameter should be taken into account (if not set then all)

      What behaviour do you see?

      • We see in the Kafka Connectors log an error message when we start a connector for schema_X with table list Table_1, Table2 
        • about schema_Y ???? --> why never set in a connector ?
        • about table_Y ??? --> why never set in a connector ?

      Do you see the same behaviour using the latest relesead Debezium version?

      (Ideally, also verify with latest Alpha/Beta/CR version)

      seems in the past that it was a WARNING message now an ERROR (not sure)

      Do you have the connector logs, ideally from start till finish?

      (You might be asked later to provide DEBUG/TRACE level log)

        "tasks": [
         

      {       "id": 0,       "state": "FAILED",       "worker_id": "hrxkfddc01.hrx.erp:8083",       "trace": "io.debezium.text.ParsingException: Trying to change column pending_form_id in hwm.activity_logs table, which does not exist. Query: CHANGE `pending_form_id` `form_id` int(11) DEFAULT NULL\n\tat io.debezium.connector.mysql.antlr.listener.AlterTableParserListener.lambda$enterAlterByChangeColumn$8(AlterTableParserListener.java:167)\n\tat io.debezium.connector.mysql.antlr.MySqlAntlrDdlParser.runIfNotNull(MySqlAntlrDdlParser.java:387)\n\tat io.debezium.connector.mysql.antlr.listener.AlterTableParserListener.enterAlterByChangeColumn(AlterTableParserListener.java:150)\n\tat io.debezium.ddl.parser.mysql.generated.MySqlParser$AlterByChangeColumnContext.enterRule(MySqlParser.java:15586)\n\tat io.debezium.antlr.ProxyParseTreeListenerUtil.delegateEnterRule(ProxyParseTreeListenerUtil.java:46)\n\tat io.debezium.connector.mysql.antlr.listener.MySqlAntlrDdlParserListener.enterEveryRule(MySqlAntlrDdlParserListener.java:89)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.enterRule(ParseTreeWalker.java:41)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:25)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)\n\tat org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:28)\n\tat io.debezium.antlr.AntlrDdlParser.parse(AntlrDdlParser.java:83)\n\tat io.debezium.relational.history.AbstractSchemaHistory.lambda$recover$3(AbstractSchemaHistory.java:174)\n\tat io.debezium.storage.kafka.history.KafkaSchemaHistory.recoverRecords(KafkaSchemaHistory.java:339)\n\tat io.debezium.relational.history.AbstractSchemaHistory.recover(AbstractSchemaHistory.java:134)\n\tat io.debezium.relational.history.SchemaHistory.recover(SchemaHistory.java:152)\n\tat io.debezium.relational.HistorizedRelationalDatabaseSchema.recover(HistorizedRelationalDatabaseSchema.java:62)\n\tat io.debezium.schema.HistorizedDatabaseSchema.recover(HistorizedDatabaseSchema.java:39)\n\tat io.debezium.connector.mysql.MySqlConnectorTask.validateAndLoadSchemaHistory(MySqlConnectorTask.java:347)\n\tat io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:102)\n\tat io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:135)\n\tat org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:232)\n\tat org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:189)\n\tat org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:238)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"     }

        ],
        "type": "source"
      }

      You can find some historical info in case: DBZ-5840

      How to reproduce the issue using our tutorial deployment?

      <Your answer>

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      <Your answer>

      Implementation ideas (optional)

      None listed schema/tables should be IGNORED or  be a WARNING but should never stop/block a connector for a specific schema and specific tables (=SINGLETON, ignore other)

       

        1. debezium connector 1.8 warn and not fail and seem default is FAIL .png
          347 kB
          Eric FRANCKX
        2. debezium connector 1.9.7 WARN message.png
          95 kB
          Eric FRANCKX
        3. debezium connector 2.0 ERROR in place of WARN message.png
          93 kB
          Eric FRANCKX
        4. debezium schema of other DB in historic of 1 specific connector for 1 Schema only.png
          31 kB
          Eric FRANCKX
        5. screenshot-1.png
          251 kB
          Eric FRANCKX
        6. screenshot-2.png
          85 kB
          Harvey Yue
        7. screenshot-3.png
          84 kB
          Harvey Yue

              Unassigned Unassigned
              eric.franckx@ngahr.com Eric FRANCKX (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: