Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8472

Connector configuration logging improvement

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Minor Minor
    • 3.0.5.Final
    • 3.0.4.Final
    • core-library
    • None

      Configuration for Debezium source connectors in production are typically large, ours are 60+ lines.

      Current implementation logs each key/value pair separately, this is inefficient and can be hard to follow.

      Current implementation is here

      Proposed solution: log connector configs in one statement as is done for Kafka, e.g:

      ConsumerConfig values: 
      	allow.auto.create.topics = true
      	auto.commit.interval.ms = 5000
      	auto.include.jmx.reporter = true
      	auto.offset.reset = earliest
      	bootstrap.servers = []
      	check.crcs = true
      	client.dns.lookup = use_all_dns_ips
      	client.id = 
      	client.rack = 
      	connections.max.idle.ms = 540000
      	default.api.timeout.ms = 60000
      	enable.auto.commit = false
      	enable.metrics.push = true
      	exclude.internal.topics = true

      Proposed implementation: use StringBuilder in BaseSourceTask.start() to build an aggregated connector configuration log message and log in one statement.

      Additional benefit: connector configuration logs ingested into DataDog are in one record.

              Unassigned Unassigned
              m8719-redhat Andrei Leibovski
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: