Details
-
Bug
-
Resolution: Not a Bug
-
Major
-
None
-
2.1.2.Final
-
None
-
False
-
None
-
False
Description
In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
Bug report
For bug reports, provide this information, please:
What Debezium connector do you use and what version?
Vitess Connector v2.1
What is the connector configuration?
Two connectors in the deployment for the same Kafka connector environment:
db1_connector =
{ topic.prefix: "dev", vitess.keyspace: "db1", table.include.list = "tab1" },
db2_connector =
{ topic.prefix: "dev", vitess.keyspace: "db2", table.include.list = "tab2" },
In Vitess, 'keyspace' has the same meaning as database name. For the above configuration, two kafka data topics will be created:
dev.db1.tab1
dev.db2.tab2
What is the captured database version and mode of depoyment?
Vitess V12 on AWS
What behaviour do you expect?
We were running with this configuration fine in debezium v1.9, after we upgraded to v2.1, we saw the below warnings in the log (repeating 12 times) and this significantly slows down connector starts up:
Vitess|dev|snapshot Unable to register metrics as an old set with the same name exists, retrying in PT5S (attempt 1 out of 12) [io.debezium.metrics.Metrics]
After investigation, we realized the problem is the two connector has the same topic.prefix, hence the same connection logical name. So the metrics for the 2nd connector cannot be created.
From the doc, it looks like 'topic.prefix' is a newly introduced required parameter for the connector in v2.1 and it is used as the logical name of the connector and it is expected this topic.prefix should be unique across all connectors.
In our configuration, the two connectors are pre-existing and the Kafka topic names are also pre-existing: dev.db1.tab1 and dev.db2.tab2.
In our configuration, each connector is pulling from its respective database and database name (keyspace name) is already part of the Kafka topic name. The two Kafka topic names are already distinct without that topic.prefix. If you want the two connectors having different topic prefixes, I will end up giving topic.prefix: dev.db1 for the 1st connector and dev.db2 for the 2nd connector. So the final two Kafka topic names will become: dev.db1.db1.tab1 and dev.db2.db2.tab1 with database name repeating twice in the topic name which is redundant and ugly. And this also breaks our existing Kafka topic names for this debezium migration.
I understand the connector's name needs to be unique but defaulting the name to the topic prefix is an over-simplication. I think we should have a separate connector config: logical.name. Connector's logical name should be set to this config value if logical.name config is present otherwise fallback to topic.prefix. Connector's logical.name needs to be unique but not topic.prefix
What behaviour do you see?
Metrics are not generated for the 2nd connector, and warning like below:
Vitess|dev|snapshot Unable to register metrics as an old set with the same name exists, retrying in PT5S (attempt 1 out of 12) [io.debezium.metrics.Metrics]
Do you see the same behaviour using the latest relesead Debezium version?
Tested on debezium v2.1
Do you have the connector logs, ideally from start till finish?
(You might be asked later to provide DEBUG/TRACE level log)
<Your answer>
How to reproduce the issue using our tutorial deployment?
<Your answer>
Feature request or enhancement
For feature requests or enhancements, provide this information, please:
Which use case/requirement will be addressed by the proposed feature?
<Your answer>
Implementation ideas (optional)
<Your answer>