Details
-
Task
-
Resolution: Done
-
Major
-
1.2.2.Final
-
None
-
False
-
False
-
Undefined
Description
On our production machine we are using the Sql Server connector with the embedded engine. For the Sql Server connector the following settings are required even though we are not using kafka: database.history.kafka.bootstrap.servers & database.history.kafka.topic. If these are not set you get an error when starting the connector.
Since these settings seem to be required, we set them as follows: database.history.kafka.bootstrap.servers=kafka:9092
database.history.kafka.topic=dbhistory.pt_survey
On the production machine port 9092 is blocked and we get the following error when starting the connector:
[WARN] 2020-10-15 08:49:13,368 org.apache.kafka.clients.ClientUtils parseAndValidateAddresses - Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka [INFO] 2020-10-15 08:49:13,368 org.apache.kafka.clients.producer.KafkaProducer close - [Producer clientId=pt_survey-dbhistory] Closing the Kafka producer with timeoutMillis = 0 ms. [DEBUG] 2020-10-15 08:49:13,368 org.apache.kafka.clients.producer.KafkaProducer close - [Producer clientId=pt_survey-dbhistory] Kafka producer has been closed [INFO] 2020-10-15 08:49:13,368 org.apache.kafka.connect.storage.FileOffsetBackingStore stop - Stopped FileOffsetBackingStore [ERROR] 2020-10-15 08:49:13,368 io.debezium.embedded.EmbeddedEngine lambda$new$0 - Unable to initialize and start connector's task class 'io.debezium.connector.sqlserver.SqlServerConnectorTask' with config: {connector.class=io.debezium.connector.sqlserver.SqlServerConnector, database.user=debezium, database.dbname=pt_survey, converter.schemas.enable=false, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, tasks.max=1, database.history.kafka.bootstrap.servers=kafka:9092, database.history.kafka.topic=dbhistory.pt_survey, database.server.name=pt_survey, database.port=1433, table.whitelist=dbo.PatientSurveyDue, offset.flush.interval.ms=5000, key.converter.schemas.enable=false, offset.storage.file.filename=pt_survey-offsets.dat, event.hubs.topic=aeh.vivify.patientsurveydue-changes-dev, database.hostname=100.67.0.12, database.password=********, poll.interval.ms=30000, name=pt_survey-connector, value.converter.schema.enable=false, max.batch.size=250, schemas.enable=false} org.apache.kafka.common.KafkaException: Failed to construct kafka producer at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298) at io.debezium.relational.history.KafkaDatabaseHistory.start(KafkaDatabaseHistory.java:235) at io.debezium.relational.HistorizedRelationalDatabaseSchema.<init>(HistorizedRelationalDatabaseSchema.java:40) at io.debezium.connector.sqlserver.SqlServerDatabaseSchema.<init>(SqlServerDatabaseSchema.java:33) at io.debezium.connector.sqlserver.SqlServerConnectorTask.start(SqlServerConnectorTask.java:82) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:101) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:758) at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:170) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:89) at org.apache.kafka.clients.ClientUtils.parseAndValidateAddresses(ClientUtils.java:48) at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:408) ... 11 more
We are assuming that we get this error because port 9092 is blocked. We could not find another setting to not require database history.
It seems that really should not have to open a port to resolve kafka brokers when they are not going to be used by embedded engine.
