Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2984

ExtractNewRecord SMT incorrectly extracts ts_ms from source info

XMLWordPrintable

    • False
    • False
    • 1.4
    • Undefined
    • Hide
      1. Set up a sink connector using the event flattening instructions, using debezium < 1.4
        1. Note - I used the s3 sink, but I imagine any sink should work
        2. include transforms.unwrap.add.fields=source.ts_ms
        3. Do not set up an "alias" for source.ts_ms (i.e. do not use source.ts_ms:EVENT_TIMESTAMP option mentioned in add.fields)
      2. read some messages from the sink location, ensure that the field is coming in as __source_ts_ms
      3. Upgrade to debezium 1.4, restart the sink connector
      4. Verify that the new messages come through with _null_ts_ms instead of }}{{_source_ts_ms
        1. (note that the above should have double underscores as the leading prefix; having issues with Jira formatting)
      Show
      Set up a sink connector using the  event flattening instructions , using debezium < 1.4 Note - I used the s3 sink, but I imagine any sink should work include transforms.unwrap.add.fields=source.ts_ms Do not set up an "alias" for source.ts_ms (i.e. do not use source.ts_ms:EVENT_TIMESTAMP option mentioned in add.fields) read some messages from the sink location, ensure that the field is coming in as __source_ts_ms Upgrade to debezium 1.4, restart the sink connector Verify that the new messages come through with _ null_ts_ms instead of }} {{_source_ts_ms (note that the above should have double underscores as the leading prefix; having issues with Jira formatting)

      I recently upgraded to debezium 1.4 in our dev environment, and noticed our field _source_ts_ms was no longer populating. After some digging, I noticed that the field was being populated as _null_ts_ms. This appears to me to be related to the addition of this line (the function called returns null for an input of ts_ms). 

       

      For me personally, this is a problem as some our upsert solution is based upon __source_ts_ms, which - since the key is now missing in the message, returns null for all new values.

              jpechane Jiri Pechanec
              p5k6@yahoo.com josh stanfield (Inactive)
              Votes:
              4 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: