Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-8082

ConverterBuilder doesn't pass Headers to be manipulated

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Minor Minor
    • 3.0.0.Beta1
    • 3.0.0.Alpha1
    • debezium-server
    • None

      Feature request or enhancement

      Debezium doesn't pass the record headers to the converter so that they can be manipulated or used as part of the converter
      Source 1 - ConverterBuilder
      Source 2 - Kafka Connect Converter

      We can pass the headers from the record.headers but they need to be casted from connect.header.Headers -> common.header.Headers.

      Which use case/requirement will be addressed by the proposed feature?

      Internally we rely on these headers in the value converter to get around the max message size of Kafka.

      Essentially we can modify the value structure and pass information in the Headers but they need to be passed through in order to then get merged into the payload by any header converter.

      Implementation ideas (optional)

      Example of what might need to be done to make this a reality: https://github.com/debezium/debezium/compare/main...ryanvanhuuksloot:debezium:rvh.example-record-headers

      Note: it is ugly and untested but hopefully it is demonstrative

      I was trying to think of a smart way to avoid a double for loop but I can't see any way because of the double cast from org.apache.kafka.connect.header.Headers -> org.apache.kafka.common.header.Headers -> List<io.debezium.engine.Header>. You can't just pass in empty headers into the key/value converter because it is possible users may want access to the headers for read purposes.

              Unassigned Unassigned
              ryanvanhuuksloot Ryan van Huuksloot (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: