-
Bug
-
Resolution: Done
-
Major
-
0.9.3.Final
-
None
Issue
When a MongoDB collection haven't had activity for a period of time an initial sync is
triggered
Context
Let's see the scenario:
- The connector is working normally and publishes the `offset` of the oplog in the kafka connect offset topic, marking the offset as `2019-03-25 10:00`
- There's no activity for more than a day in the monitored collection
- oplog is a buffer which can last less or longer depending on the configured size, but for our example it's 24 hours
- Right now is `2019-03-26 17:00`
- The connector task is restarted or rebalanced, it tries to reach the offset `2019-03-25 10:00` but it's no longer available, because the oldest available right now is `2019-03-25 17:00`
- The connector causes an initial sync since it couldn't find the oplog offset
This happens because since the collection haven't had any update for longer than 24 hours there was nothing updating the offset in the KC offset topic.
Suggestion
Often (configured time) update the current oplog cursor offset even if there are no events for the specific monitored collection
Our case
It's not a "real" issue since this is happening in our staging cluster due to inactivity, it won't ever happen in our production cluster, it's just an inconvenience which made me feel insecure about the production state and that's why I had to research on it
Implementation
I'd like to hear possible implementation strategies for this, an suggestions?