Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1781

Add support for update events for sharded MongoDB collections

    XMLWordPrintable

    Details

      Description

      "Update" change events for MongoDB do not include all of the fields needed when the event is from a sharded collection that uses multiple fields for the shard key. Here is what an "update" oplog entry looks like where the shard key is made up of dc, acctId and _id:

      {
          "ts" : Timestamp(1570559743, 2),
          "t" : NumberLong(180),
          "h" : NumberLong(-2190864266344322319),
          "v" : 2,
          "op" : "u",
          "ns" : "Asset",
          "ui" : UUID("8667a4d5-051f-4257-bfa4-451b6836b651"),
          "o2" : {
              "dc" : "west",
              "acctId" : NumberLong(1234567),
              "_id" : NumberLong(902345287483)
          },
          "wall" : ISODate("2019-10-08T18:35:43.431Z"),
          "o" : {
              "$v" : 1,
              "$set" : {...}
          }
      }
      

      The current code will just use the _id field as the key so the other two fields will be lost:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "902345287483"
        }
      }
      

      I propose that when multiple fields are in o2 the key should use all of them as the id like so:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "{ \"dc" : \"west\" , \"acctId\" : 1234567, \"_id\" : 902345287483}"
        }
      }
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            jgraf50 John Graf (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: