Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-5995

MongoDB Transformation Stage

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Unresolved
    • Icon: Major Major
    • 2.2-backlog
    • None
    • mongodb-connector
    • None
    • False
    • None
    • False

      Currently, I am using the MongoDB Kafka Source Connector to stream the event in my local database to another MongoDB instance in the cloud.  We want to replace the remote DB by a Postgresql instance and to do it, we want to use the Debezium Mongodb source connector with the Debezium sink connector for Postgresql.

      My source connector has an aggregation pipeline to modify the payload to another schema before putting it into Kafka. The example below shows how to achieve such behavior:

      ```
      {
        "name": "metrics",
        "config": {
          "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
          "connection.uri": "mongodb://172.0.0.2:27017",
          "database": "metrics",
          "collection": "metrics",
          "copy.existing": "true",
          "pipeline": "[{"$match": { "$and": [

      {"operationType": "insert"}

      , {"fullDocument.eventId": 1404 }] } }]"
        }
      }
      ```

      It would be great to Debezium source connector for MongoDB support this kind of aggregation pipeline to transform each streamed document. In another words, The feature consists in a SMT using mongoDB aggregation stage.

      MongodDB Documentation Example

      Thanks

              Unassigned Unassigned
              paulopereira-1 Paulo Pereira (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: