Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6867

Streaming aggregation pipeline broken for combination of database filter and signal collection

    XMLWordPrintable

Details

    • Critical

    Description

      Related Zulip discussion: https://debezium.zulipchat.com/#narrow/stream/302529-community-general/topic/Infrequent.20traffic.20MongoDB.20not.20streaming.20change.20events

      Problematic code:
      https://github.com/debezium/debezium/blob/main/debezium-connector-mongodb/src/main/java/io/debezium/connector/mongodb/ChangeStreamPipelineFactory.java#L119

      return andFilters(
                      dbFilters,
                      orFilters(
                              includedSignalCollectionFilters,
                              collectionsFilters));
      

      When database.include.list is specified together with signal collection the aggregation match filter above will match ONLY the signal collection.

      The signal collection should be matched ALWAYS. Thus the filter should match even's if they are coming from the signal collection OR they match the combination of database and collection filter (that is the signal collection should not be considered part of the collection filters).

      The correct code should look like this

      return orFilters(
                      includedSignalCollectionFilters,
                      andFilters(
                              dbFilters,
                              collectionsFilters
                      ));
      

      Attachments

        Issue Links

          Activity

            People

              jcechace@redhat.com Jakub Čecháček
              jcechace@redhat.com Jakub Čecháček
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: