Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-1198

When a MongoDB collection haven't had activity for a period of time an initial sync is triggered

    Details

      Description

      Issue

      When a MongoDB collection haven't had activity for a period of time an initial sync is
      triggered

      Context

      Let's see the scenario:

      • The connector is working normally and publishes the `offset` of the oplog in the kafka connect offset topic, marking the offset as `2019-03-25 10:00`
      • There's no activity for more than a day in the monitored collection
      • oplog is a buffer which can last less or longer depending on the configured size, but for our example it's 24 hours
      • Right now is `2019-03-26 17:00`
      • The connector task is restarted or rebalanced, it tries to reach the offset `2019-03-25 10:00` but it's no longer available, because the oldest available right now is `2019-03-25 17:00`
      • The connector causes an initial sync since it couldn't find the oplog offset

      This happens because since the collection haven't had any update for longer than 24 hours there was nothing updating the offset in the KC offset topic.

      Suggestion

      Often (configured time) update the current oplog cursor offset even if there are no events for the specific monitored collection

      Our case

      It's not a "real" issue since this is happening in our staging cluster due to inactivity, it won't ever happen in our production cluster, it's just an inconvenience which made me feel insecure about the production state and that's why I had to research on it

      Implementation

      I'd like to hear possible implementation strategies for this, an suggestions?

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                jpechanec Jiri Pechanec
                Reporter:
                renatomefi Renato Mefi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: