Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3804

Debezium CDC with Apicurio Schema Registry

XMLWordPrintable

    • False
    • False
    • Undefined

      Hello
       
      I am creating a proof of concept using apicurio schema registry (JPA) and debezium CDC docker images.
       
      I have:
         1.- zookeeper configured
         2.- kafka configured 
         3.- mysql configured
         4.- postgres for apicurio regstry jpa
         5.- debezium connect for mysql
         6.- apicurio registry jpa
       
      All these dockers are up and running successfully.
       
      I have created a mysql database and automatically I can see the schemas in apicurio but if I run an alter table command this is not propagate to the apicurio schema registry (I am able to see the alter in kafka but not in the apicurio ui).
       
      I have following mysql connect properties file: 
      {{    "name": "classicmodels-connector"   ,"config": {
               "connector.class": "io.debezium.connector.mysql.MySqlConnector"
              ,"tasks.max": "1"
              ,"database.hostname": "mysql"        ,"database.port": "3306"        ,"database.user": "root"        ,"database.password": "FYK11IBMgVocfNzyQILMVlY22NYh80PUxQpP+xou9R0="        ,"database.server.id": "11111111"        ,"database.server.name": "mysql"        ,"database.include.list": "classicmodels"        ,"database.allowPublicKeyRetrieval":"true"        ,"database.history.kafka.bootstrap.servers": "kafka1:9091,kafka2:9092,kafka3:9093"        ,"database.history.kafka.topic": "schema-changes.classicmodels"
              ,"transforms": "route"        ,"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter"        ,"transforms.route.regex": "([^.])\\.([^.])
      .([^.]+)"        ,"transforms.route.replacement": "$3"     ,"transforms.InsertField.type": "org.apache.kafka.connect.transforms.InsertField$Value"     ,"transforms.ReplaceField.type": "org.apache.kafka.connect.transforms.ReplaceField$Value"
      ,"include.schema.changes": "true"
              ,"auto.create": "true"        ,"auto.evolve": "true"
              ,"key.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"key.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"key.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"         ,"key.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"         ,"key.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"key.converter.apicurio.registry.auto-register": "true"        ,"key.converter.apicurio.registry.find-latest": "true"        ,"key.converter.apicurio.registry.as-confluent": "true"
              ,"value.converter": "io.apicurio.registry.utils.converter.AvroConverter"        ,"value.converter.apicurio.registry.url": "http://${DOCKER_HOST_IP}:8181/api"        ,"value.converter.apicurio.registry.converter.serializer": "io.apicurio.registry.utils.serde.AvroKafkaSerializer"        ,"value.converter.apicurio.registry.converter.deserializer": "io.apicurio.registry.utils.serde.AvroKafkaDeserializer"        ,"value.converter.apicurio.registry.global-id": "io.apicurio.registry.utils.serde.strategy.GetOrCreateIdStrategy"        ,"value.converter.apicurio.registry.auto-register": "true"        ,"value.converter.apicurio.registry.find-latest": "true"        ,"value.converter.apicurio.registry.as-confluent": "true"
      }}
       
      As this part doesn't work I didn't create the sink JDBC connector to complete the PoC.
       
      Let me know if you need docker compose file I am using.
       
       Note: I am no able to find a full example of the connect properties file neither documentation of the connector properties file.
       
      Thanks in advance
       
      Regards

            jpechane Jiri Pechanec
            carlos.de.prada Carlos Sierra Fernández de Prada (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: