-
Bug
-
Resolution: Done
-
Major
-
2.3.3.Final, 2.4.0.Beta1
-
None
-
False
-
None
-
False
-
-
-
Important
The first stage of our internal aggregation pipeline manipulates the entire change event document in order to create the namespace filed (since our collection include/exclude list operate over fully classified collection name)
{ "$replaceRoot": { "newRoot": { "event": "$$ROOT", "namespace": { "$concat": [ "$ns.db", ".", "$ns.coll" ] }}}},
Combined with the fact that our change stream is deployment / database scoped this has an unfortunate side effect of failing when the full change event violates the maximum BSON size regardless if the document should be filtered out based on include / exclude list properties.
A seemingly robust solution is adding the following stage as the first to the effective aggregation pipeline
[
{
"$match": {
"$and": [
{
"$expr": {
"$lte": [
{
"$bsonSize": "$fullDocument"
},
8000
]
}
},
{
"$expr": {
"$lte": [
{
"$bsonSize": "$fullDocumentBeforeChange"
},
8000
]
}
}
]
}
}
]
Where 8000 is an arbitrary configurable value
- is blocked by
-
DBZ-6872 Configurable order of user defined and internal aggregation pipeline
-
- Closed
-
- is documented by
-
DBZ-6883 Document cursor pipeline ordering and oversize document handling mode
-
- Closed
-
- links to
-
RHEA-2024:129636
Red Hat build of Debezium 2.5.4 release
