Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-865

Error when MongoDB collection contains characters not compatible with kafka topic naming

    XMLWordPrintable

Details

    • Hide

      1. Create a mongo collection with name "my@collection", for instance, and add a record.
      2. Start mongo source connector that reads from this mongo table

      Observed result:

      • data is not written to kafka.
      • Endless logs: 
        [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
      Show
      1. Create a mongo collection with name "my@collection", for instance, and add a record. 2. Start mongo source connector that reads from this mongo table Observed result: data is not written to kafka. Endless logs: [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)

    Description

      If a mongo collection contains characters that are not compatible with the Kafka topic naming conventions (it contains a character other than ASCII alphanumerics, '.', '_' and '-'), the connector will fail when trying to create and write the data down to a topic.
      Moreover, the connecter is unable to stop throwing the error, continuing retrying and ending up filling up the logs.

      Suggested fix:
      The source connector should be able to identify invalid characters for kafka topics on the mongo collection and skip them when creating the topic.

      Extra notes:
      Transforms did not seem to do as work around. Topic with original collection name is still tried to be created.

      transforms=Reroute
transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
transforms.Reroute.topic.regex=(.*)my@collection(.*)
transforms.Reroute.topic.replacement=$1my_collection

      Attachments

        Issue Links

          is related to

          Enhancement - An enhancement or refactoring of existing functionality DBZ-878 MySQL identifier with dot or space could not be parsed

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              gunnar.morling Gunnar Morling
              ruileal_jira Rui Leal (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: