Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1781

Add support for update events for sharded MongoDB collections

    Details

      Description

      "Update" change events for MongoDB do not include all of the fields needed when the event is from a sharded collection that uses multiple fields for the shard key. Here is what an "update" oplog entry looks like where the shard key is made up of dc, acctId and _id:

      {
          "ts" : Timestamp(1570559743, 2),
          "t" : NumberLong(180),
          "h" : NumberLong(-2190864266344322319),
          "v" : 2,
          "op" : "u",
          "ns" : "Asset",
          "ui" : UUID("8667a4d5-051f-4257-bfa4-451b6836b651"),
          "o2" : {
              "dc" : "west",
              "acctId" : NumberLong(1234567),
              "_id" : NumberLong(902345287483)
          },
          "wall" : ISODate("2019-10-08T18:35:43.431Z"),
          "o" : {
              "$v" : 1,
              "$set" : {...}
          }
      }
      

      The current code will just use the _id field as the key so the other two fields will be lost:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "902345287483"
        }
      }
      

      I propose that when multiple fields are in o2 the key should use all of them as the id like so:

      {
        "schema": {
          "type": "struct",
          "fields": [
            {
              "type": "string",
              "optional": false,
              "field": "id"
            }
          ],
          "optional": false,
          "name": "Asset.Key"
        },
        "payload": {
          "id": "{ \"dc" : \"west\" , \"acctId\" : 1234567, \"_id\" : 902345287483}"
        }
      }
      

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jgraf50 John Graf
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: