Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-865

Error when MongoDB collection contains characters not compatible with kafka topic naming

XMLWordPrintable

    • Hide

      1. Create a mongo collection with name "my@collection", for instance, and add a record.
      2. Start mongo source connector that reads from this mongo table

      Observed result:

      • data is not written to kafka.
      • Endless logs: 
        [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
[2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)
      Show
      1. Create a mongo collection with name "my@collection", for instance, and add a record. 2. Start mongo source connector that reads from this mongo table Observed result: data is not written to kafka. Endless logs: [2018-08-17 09:20:39,685] WARN [Producer clientId=producer-46] Error while fetching metadata with correlation id 308131 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-52] Error while fetching metadata with correlation id 307868 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,686] WARN [Producer clientId=producer-43] Error while fetching metadata with correlation id 329832 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246) [2018-08-17 09:20:39,712] WARN [Producer clientId=producer-49] Error while fetching metadata with correlation id 307934 : {testnl.testnl.my@collection=INVALID_TOPIC_EXCEPTION} (org.apache.kafka.clients.NetworkClient:246)

      If a mongo collection contains characters that are not compatible with the Kafka topic naming conventions (it contains a character other than ASCII alphanumerics, '.', '_' and '-'), the connector will fail when trying to create and write the data down to a topic.
      Moreover, the connecter is unable to stop throwing the error, continuing retrying and ending up filling up the logs.

      Suggested fix:
      The source connector should be able to identify invalid characters for kafka topics on the mongo collection and skip them when creating the topic.

      Extra notes:
      Transforms did not seem to do as work around. Topic with original collection name is still tried to be created.

      transforms=Reroute
transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
transforms.Reroute.topic.regex=(.*)my@collection(.*)
transforms.Reroute.topic.replacement=$1my_collection

              gunnar.morling Gunnar Morling
              ruileal_jira Rui Leal (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: