-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
None
-
False
-
None
-
False
-
-
I think there might be an issue in the behavior of the Mongo DB connector regarding fetching metadata like the list of matching tables. Currently the user can provide a connection string for the Mongo DB connector that enforces using a specific replica node or a set of replicas that match certain replica set tags instead of connection to the primary node.
But reading cluster/replica-set metadata like the list of available collections can lead to confusion for users when the replica node / secondary is lagging behind replication and might not be correct at all after certain events (like network portioning etc).
When a replica node falls behind in replicating changes from the primary it might:
- have missed or is not up-to-date with latest deletes of collections
- have missed or is not up-to-date with newly create collections
- after a network partition a replica node could even become a / elect as (single) primary node and local data could really become off from the rest of the cluster, then the separated replica could even have new collections created which are not available at all on the rest of the cluster
- when returning to normal operation the node will drop those collections and data created during a the partition event with a "rollback" to re-sync to the original primary node
This can also cause confusing situations like:
- Kafka Connect (with auto topic-creation enabled) creates topics for collections that were removed from the primary node already quite some time ago, but are not yet removed from the respective replica, but data never arrive because the collections are also not in use anymore at all (by applications etc). No data arrive in that topic although it looks like it is required.
- TBD
If this issue can be confirmed I think we should change this behavior to load metadata from the primary node, even when streaming from a secondary/replica node was configured to avoid those inconsistencies and confusing situations.
- blocks
-
DBZ-7166 Align Mongo DB metadata retrieval between REST extension (when loading filter results) and the connector
-
- Open
-