Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6170

Connect and stream from sharded clusters through mongos instances

    XMLWordPrintable

Details

    • Enhancement
    • Resolution: Done
    • Major
    • 2.2.0.Beta1
    • 2.2.0.Alpha2
    • mongodb-connector
    • None
    • False
    • None
    • False

    Description

      Currently for sharded clusters Debezium opens a connection to each shard's replica set directly for snapshot and streaming.

      While this approach should allow for a higher throughput due to utilised parallelism,[ it is not recommended by MongoDb|https://www.mongodb.com/docs/manual/sharding/#connecting-to-a-sharded-cluster]

      We should provide a way to utilise proper connection via mongos instance. However this solution should be backwards compatible – at least for the lifespan on 2.x release line.

      Additionally this will allow support of incremental snapshotting for sharded deployments as well as total ordering of captured change events.

      Attachments

        Issue Links

          depends on

          Task - Represents a small unit of work that is not end-user facing. DBZ-6032 Refactor connection management for mongodb connector

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-5987 Refactor snapshotting to use change streams instead of oplog

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          incorporates

          Task - Represents a small unit of work that is not end-user facing. DBZ-4427 Complete MongoDB incremental snapshotting implementation

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) DBZ-5590 Capture evenents in order across mongodb shards

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link RHEA-2023:120698 Red Hat build of Debezium 2.3.4 release

          Activity

            People

              jcechace@redhat.com Jakub Čecháček
              jcechace@redhat.com Jakub Čecháček
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: