Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4083

CloudEventsConverter omits payload data of deleted documents

    XMLWordPrintable

Details

    Description

      Observation

      When using Debezium connector for MongoDB with CloudEventsConverter, change events related to delete operations do not include any payload data for deleted records. In continuous replication use-cases, it can be very annoying to get notified about deleted records without being able to identify them.

      Note that there is no issue with the regular converter because FILTER field contains the JSON string representation of the MongoDB selection criteria that was used to identify the document to be deleted, ex.

      "filter":"{\"_id\": {\"$oid\": \"60db4680f661ef1f05c27642\"}}"
      

      Analysis

      When using Debezium connector for PostgreSQL with CloudEventsConverter, change events related to delete operations include BEFORE field, which is not as optimal (as FILTER field mentioned above with regards to message size):

          public PostgresRecordParser(Schema schema, Struct record) {
              super(schema, record, Envelope.FieldName.BEFORE, Envelope.FieldName.AFTER);
          }
      

      When using Debezium connector for MongoDB with CloudEventsConverter, change events only include payload AFTER:

          public MongoDbRecordParser(Schema schema, Struct record) {
              super(schema, record, Envelope.FieldName.AFTER, "patch");
          }
      

      CloudEvent message format is expected by our consumer.

      Suggestion

      Either include regular FILTER field to CloudEvent payload, or pass it to the BEFORE field.

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            ju28482 Ju Clarysse (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: