-
Feature Request
-
Resolution: Done
-
Minor
-
0.10.0.CR1
We have a use case that requires us to flatten the events produced by the Postgres connector. Similar to the "add.source.fields" setting that was added to the ExtractNewRecordState transform, it would be great if we could get the "op" value in the event to be added to the flattened record. Similar to how you have the "operation.header" setting in the ExtractNewDocumentState transform
[DBZ-1452] Add ability to insert fields from op field in ExtractNewRecordState SMT
Jiri Pechanec
added a comment - Released Jiri Pechanec
added a comment - gunnar.morling I am saying that we should provide the similar powerful conf as we do in cas of EventRouter. Just in case of this it means that source and envelope elements could be extracted and recorded either in headers or in fields.
Jiri Pechanec
added a comment - gunnar.morling I am saying that we should provide the similar powerful conf as we do in cas of EventRouter. Just in case of this it means that source and envelope elements could be extracted and recorded either in headers or in fields. 
Gunnar Morling
added a comment - jpechane are you saying then that the extractable "op" fields should also be supported to go into header properties?
Gunnar Morling
added a comment - jpechane are you saying then that the extractable "op" fields should also be supported to go into header properties? 
Andre Glenn (Inactive)
added a comment - gunnar.morling - I see what you're saying, and No, having the op in a Message Header doesn't work. Or at least I assume that the JDBCSinkConnector doesn't understand Kafka Message Headers.
jpechane - I think that I see what you are referring to. Yes, I can move that code from debezium-core/src/main/java/io/debezium/transforms/outbox to the ExtractNewRecordState code in my PR once I get to this task.
Andre Glenn (Inactive)
added a comment - gunnar.morling - I see what you're saying, and No, having the op in a Message Header doesn't work. Or at least I assume that the JDBCSinkConnector doesn't understand Kafka Message Headers.
jpechane - I think that I see what you are referring to. Yes, I can move that code from debezium-core/src/main/java/io/debezium/transforms/outbox to the ExtractNewRecordState code in my PR once I get to this task. Jiri Pechanec
added a comment - Hi, how about porting of io.debezium.transforms.outbox.EventRouterConfigDefinition.FIELDS_ADDITIONAL_PLACEMENT feature to this SMT too?
This would allow extraction of arbitrary envelope and source field and map it to either header or fields in the new messages?
Jiri Pechanec
added a comment - Hi, how about porting of io.debezium.transforms.outbox.EventRouterConfigDefinition.FIELDS_ADDITIONAL_PLACEMENT feature to this SMT too?
This would allow extraction of arbitrary envelope and source field and map it to either header or fields in the new messages? Ha, I initially thought the same, but then figured (maybe incorrectly) that you'd want something different. Currently, you can propagate the operation as a message header, whereas, IIUC, you want to propagate it as a field in the emitted record. This is something you cannot do atm., only the fields from the source struct can be added as fields in the record.
Gunnar Morling
added a comment - - edited Ha, I initially thought the same, but then figured (maybe incorrectly) that you'd want something different. Currently, you can propagate the operation as a message header , whereas, IIUC, you want to propagate it as a field in the emitted record. This is something you cannot do atm., only the fields from the source struct can be added as fields in the record. Andre Glenn (Inactive)
added a comment - Say gunnar.morling, I'm a bit confused. I'm looking at what is now the 0.10 doc page for the ExtractNewRecordState SMT and it appears that functionality to include the "op" in the output of that SMT has been added to the 0.10 version. https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html
This seems to be what I asked for in the Feature Request that I made.
So I'm not clear what it would be necessary for me to change if the functionality is already there in the 0.10 version. Can you clarify?
Andre Glenn (Inactive)
added a comment - Say gunnar.morling , I'm a bit confused. I'm looking at what is now the 0.10 doc page for the ExtractNewRecordState SMT and it appears that functionality to include the "op" in the output of that SMT has been added to the 0.10 version. https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html
This seems to be what I asked for in the Feature Request that I made.
So I'm not clear what it would be necessary for me to change if the functionality is already there in the 0.10 version. Can you clarify? Gunnar Morling
added a comment - Thanks a lot!
The code is at https://github.com/debezium/debezium/tree/master/debezium-core/src/main/java/io/debezium/transforms right
Yes, exactly.
Gunnar Morling
added a comment - Thanks a lot!
The code is at https://github.com/debezium/debezium/tree/master/debezium-core/src/main/java/io/debezium/transforms right
Yes, exactly. Andre Glenn (Inactive)
added a comment - Sure gunnar.morling, I can probably do that. I am somewhat familiar with Java but I haven't looked at any Java code in over 5 years.
However, looking at the code for the Transform, it seems pretty straightforward.
I've never worked with JUnit but I will use the existing tests as a guideline.
The code is at https://github.com/debezium/debezium/tree/master/debezium-core/src/main/java/io/debezium/transforms right?
Andre Glenn (Inactive)
added a comment - Sure gunnar.morling , I can probably do that. I am somewhat familiar with Java but I haven't looked at any Java code in over 5 years.
However, looking at the code for the Transform, it seems pretty straightforward.
I've never worked with JUnit but I will use the existing tests as a guideline.
The code is at https://github.com/debezium/debezium/tree/master/debezium-core/src/main/java/io/debezium/transforms right? 
Released