Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3804

Debezium CDC with Apicurio Schema Registry

XMLWordPrintable

    • False
    • False
    • Undefined

      Hello
       
      I am creating a proof of concept using apicurio schema registry (JPA) and debezium CDC docker images.
       
      I have:
         1.- zookeeper configured
         2.- kafka configured 
         3.- mysql configured
         4.- postgres for apicurio regstry jpa
         5.- debezium connect for mysql
         6.- apicurio registry jpa
       
      All these dockers are up and running successfully.
       
      I have created a mysql database and automatically I can see the schemas in apicurio but if I run an alter table command this is not propagate to the apicurio schema registry (I am able to see the alter in kafka but not in the apicurio ui).
       
      I have following mysql connect properties file: 
      {{    "name": "classicmodels-connector"   ,"config": {
               "connector.class": "io.debezium.connector.mysql.MySqlConnector"
              ,"tasks.max": "1"
              ,"database.hostname": "mysql"        ,"database.port": "3306"        ,"database.user": "root"        ,"database.password": "FYK11IBMgVocfNzyQILMVlY22NYh80PUxQpP+xou9R0="        ,"database.server.id": "11111111"        ,"database.server.name": "mysql"        ,"database.include.list": "classicmodels"        ,"database.allowPublicKeyRetrieval":"true"        ,"database.history.kafka.bootstrap.servers": "kafka1:9091,kafka2:9092,kafka3:9093"        ,"database.history.kafka.topic": "schema-changes.classicmodels"
              ,"transforms": "route"        ,"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter"        ,"transforms.route.regex": "([^.])\\.([^.])
      .([^.]+)"        ,"transforms.route.replacement": "$3"     ,"transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value"     ,"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value"
      ,"include.schema.changes": "true"
              ,"auto.create": "true"        ,"auto.evolve": "true"
              ,"key.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"key.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"key.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"         ,"key.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"         ,"key.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"key.converter.apicurio.registry.auto-register": "true"        ,"key.converter.apicurio.registry.find-latest": "true"        ,"key.converter.apicurio.registry.as-confluent": "true"
              ,"value.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"value.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"value.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"        ,"value.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"        ,"value.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"value.converter.apicurio.registry.auto-register": "true"        ,"value.converter.apicurio.registry.find-latest": "true"        ,"value.converter.apicurio.registry.as-confluent": "true"
      }}
       
      As this part doesn't work I didn't create the sink JDBC connector to complete the PoC.
       
      Let me know if you need docker compose file I am using.
       
       Note: I am no able to find a full example of the connect properties file neither documentation of the connector properties file.
       
      Thanks in advance
       
      Regards

        1. Debezium CDC.docx
          987 kB
          Carlos Sierra Fernández de Prada

            jpechane Jiri Pechanec
            carlos.de.prada Carlos Sierra Fernández de Prada (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: