Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3804

Debezium CDC with Apicurio Schema Registry

    XMLWordPrintable

Details

    • False
    • False
    • Undefined

    Description

      Hello
       
      I am creating a proof of concept using apicurio schema registry (JPA) and debezium CDC docker images.
       
      I have:
         1.- zookeeper configured
         2.- kafka configured 
         3.- mysql configured
         4.- postgres for apicurio regstry jpa
         5.- debezium connect for mysql
         6.- apicurio registry jpa
       
      All these dockers are up and running successfully.
       
      I have created a mysql database and automatically I can see the schemas in apicurio but if I run an alter table command this is not propagate to the apicurio schema registry (I am able to see the alter in kafka but not in the apicurio ui).
       
      I have following mysql connect properties file: 
      {{    "name": "classicmodels-connector"   ,"config": {
               "connector.class": "io.debezium.connector.mysql.MySqlConnector"
              ,"tasks.max": "1"
              ,"database.hostname": "mysql"        ,"database.port": "3306"        ,"database.user": "root"        ,"database.password": "FYK11IBMgVocfNzyQILMVlY22NYh80PUxQpP+xou9R0="        ,"database.server.id": "11111111"        ,"database.server.name": "mysql"        ,"database.include.list": "classicmodels"        ,"database.allowPublicKeyRetrieval":"true"        ,"database.history.kafka.bootstrap.servers": "kafka1:9091,kafka2:9092,kafka3:9093"        ,"database.history.kafka.topic": "schema-changes.classicmodels"
              ,"transforms": "route"        ,"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter"        ,"transforms.route.regex": "([^.])\\.([^.])
      .([^.]+)"        ,"transforms.route.replacement": "$3"     ,"transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value"     ,"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value"
      ,"include.schema.changes": "true"
              ,"auto.create": "true"        ,"auto.evolve": "true"
              ,"key.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"key.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"key.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"         ,"key.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"         ,"key.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"key.converter.apicurio.registry.auto-register": "true"        ,"key.converter.apicurio.registry.find-latest": "true"        ,"key.converter.apicurio.registry.as-confluent": "true"
              ,"value.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"value.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"value.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"        ,"value.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"        ,"value.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"value.converter.apicurio.registry.auto-register": "true"        ,"value.converter.apicurio.registry.find-latest": "true"        ,"value.converter.apicurio.registry.as-confluent": "true"
      }}
       
      As this part doesn't work I didn't create the sink JDBC connector to complete the PoC.
       
      Let me know if you need docker compose file I am using.
       
       Note: I am no able to find a full example of the connect properties file neither documentation of the connector properties file.
       
      Thanks in advance
       
      Regards

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            carlos.de.prada Carlos Sierra Fernández de Prada (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: