Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1781

Add support for update events for sharded MongoDB collections

    XMLWordPrintable

Details

    Description

      "Update" change events for MongoDB do not include all of the fields needed when the event is from a sharded collection that uses multiple fields for the shard key. Here is what an "update" oplog entry looks like where the shard key is made up of dc, acctId and _id:

      {
          "ts" : Timestamp(1570559743, 2),
          "t" : NumberLong(180),
          "h" : NumberLong(-2190864266344322319),
          "v" : 2,
          "op" : "u",
          "ns" : "Asset",
          "ui" : UUID("8667a4d5-051f-4257-bfa4-451b6836b651"),
          "o2" : {
              "dc" : "west",
              "acctId" : NumberLong(1234567),
              "_id" : NumberLong(902345287483)
          },
          "wall" : ISODate("2019-10-08T18:35:43.431Z"),
          "o" : {
              "$v" : 1,
              "$set" : {...}
          }
      }
      

      The current code will just use the _id field as the key so the other two fields will be lost:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "902345287483"
        }
      }
      

      I propose that when multiple fields are in o2 the key should use all of them as the id like so:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "{ \"dc" : \"west\" , \"acctId\" : 1234567, \"_id\" : 902345287483}"
        }
      }
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            jgraf50 John Graf (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: