Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2665

Explicitly mention Debezium Engine database history config for different connectors

XMLWordPrintable

    • Icon: Task Task
    • Resolution: Done
    • Icon: Major Major
    • 1.5.0.CR1
    • 1.2.2.Final
    • documentation
    • None

      On our production machine we are using the Sql Server connector with the embedded engine. For the Sql Server connector the following settings are required even though we are not using kafka: database.history.kafka.bootstrap.servers & database.history.kafka.topic.  If these are not set you get an error when starting the connector.

      Since these settings seem to be required, we set them as follows: database.history.kafka.bootstrap.servers=kafka:9092
      database.history.kafka.topic=dbhistory.pt_survey

      On the production machine port 9092 is blocked and we get the following error when starting the connector:

      [WARN] 2020-10-15 08:49:13,368 org.apache.kafka.clients.ClientUtils parseAndValidateAddresses - Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[INFO] 2020-10-15 08:49:13,368 org.apache.kafka.clients.producer.KafkaProducer close - [Producer clientId=pt_survey-dbhistory] Closing the Kafka producer with timeoutMillis = 0 ms.
[DEBUG] 2020-10-15 08:49:13,368 org.apache.kafka.clients.producer.KafkaProducer close - [Producer clientId=pt_survey-dbhistory] Kafka producer has been closed
[INFO] 2020-10-15 08:49:13,368 org.apache.kafka.connect.storage.FileOffsetBackingStore stop - Stopped FileOffsetBackingStore
[ERROR] 2020-10-15 08:49:13,368 io.debezium.embedded.EmbeddedEngine lambda$new$0 - Unable to initialize and start connector's task class 'io.debezium.connector.sqlserver.SqlServerConnectorTask' with config: {connector.class=io.debezium.connector.sqlserver.SqlServerConnector, database.user=debezium, database.dbname=pt_survey, converter.schemas.enable=false, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, tasks.max=1, database.history.kafka.bootstrap.servers=kafka:9092, database.history.kafka.topic=dbhistory.pt_survey, database.server.name=pt_survey, database.port=1433, table.whitelist=dbo.PatientSurveyDue, offset.flush.interval.ms=5000, key.converter.schemas.enable=false, offset.storage.file.filename=pt_survey-offsets.dat, event.hubs.topic=aeh.vivify.patientsurveydue-changes-dev, database.hostname=100.67.0.12, database.password=********, poll.interval.ms=30000, name=pt_survey-connector, value.converter.schema.enable=false, max.batch.size=250, schemas.enable=false}
org.apache.kafka.common.KafkaException: Failed to construct kafka producer
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
	at io.debezium.relational.history.KafkaDatabaseHistory.start(KafkaDatabaseHistory.java:235)
	at io.debezium.relational.HistorizedRelationalDatabaseSchema.<init>(HistorizedRelationalDatabaseSchema.java:40)
	at io.debezium.connector.sqlserver.SqlServerDatabaseSchema.<init>(SqlServerDatabaseSchema.java:33)
	at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:82)
	at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:101)
	at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:758)
	at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:170)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89)
	at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:408)
	... 11 more

      We are assuming that we get this error because port 9092 is blocked. We could not find another setting to not require database history.

      It seems that really should not have to open a port to resolve kafka brokers when they are not going to be used by embedded engine.

              jpechane Jiri Pechanec
              jonshuler Jon Shuler (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: