Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-58

Improve logging messages to provide connector task context

    XMLWordPrintable

Details

    Description

      The logging messages of Kafka Connect and our connector(s) can be quite difficult to follow to understand what is going on. The Connector task should use mapped diagnostic contexts, or MDC, to provide more valuable context for each log message, especially in situations where multiple connectors (of various types) are running in the same Kafka Connect service. MDC parameters can be set on a thread, and then all log messages written by components running on that thread will inherit the same context. Then, every message written to the log should include these parameters (assuming the logging configuration is configured to use them).

      I propose that Debezium adopt three MDC parameters as "standard":

      • dbz.connectorType - A short alias for the type of connector. For example, mysql, mongo, postgres, etc. All threads associated with a connector would use the same value.
      • dbz.connectorName - The name of the connector as defined in the connector's configuration. For example products, serverA, etc. All threads associated with a connector would use the same value.
      • dbz.connectorContext - A short name for a separate thread running within the connector's task. For example, main for the SourceTask implementation class, or the thread name for threads that each handle multiple source resources (e.g., tables, collections, etc.). In some cases when a thread is allocated for a specific resource (e.g., table or collection), the name of that resource could be used instead. Each thread associated with a connector would use a distinct value.

      These properties can then be used in the logging configuration. For example, in Log4J you can use %X{dbz.connectorType}, %X{dbz.connectorName}, and %X{dbz.contextName} in the format statement for a logger. Debezium-supplied logging configurations would use these to help color and identify the different logging statements.

      Attachments

        Activity

          People

            rhauch Randall Hauch (Inactive)
            rhauch Randall Hauch (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: