Details
-
Feature Request
-
Resolution: Unresolved
-
Minor
-
None
-
None
-
None
-
False
-
None
-
False
-
0
-
0%
Description
Feature request or enhancement
Which use case/requirement will be addressed by the proposed feature?
Currently users can learn about what Debezium and the connector are doing through the logs. It would be handy, however, if such information can be exposed through an API.
The context for this feature request is that I work for Airbyte. We embed Debezium in our application and use it in a non-conventional way. Usually Debezium is always on. But we have turn it on and off periodically. Consequently, it is hard to know programmatically what Debezium is doing and when is the best time to turn it off. For example, when no data is coming from the Debezium engine, there are multiple possibilities: 1) there is no change in the database; 2) Debezium is running snapshot for new tables; 3) Debezium is going through change logs that have already been processed. It is proper to shut down the engine for the first scenario, but not for the other two. If an API can tell us what Debezium and the connector is doing, that would be very helpful for our use case and other use cases like monitoring and debugging programmatically.
I have searched the backlog and did not find any existing issue coving this. But it is possible that I was not using the best search term. Please let me know if this is a duplication.
Please also let me know if such an API makes sense. We can work on the implementation if that's the case.
Implementation ideas (optional)
Add a DebeziumActivity class with two fields:
public enum DebeziumState { UNSTARTED, STARTING, SNAPSHOTTING, PROCESSING_RECORDS, WAITING_FOR_RECORDS, STOPPED } public class DebeziumActivity { private final DebeziumState state; private final Optional<String> message; }
The state enum represents what Debezium and the connector is doing at the moment. The optional message provides extra information if there is any.
In DebeziumEngine, add a new StateReporter interface:
public static interface StateReporter { default Optional<DebeziumState> hasNewState() { return Optional.empty(); } }
By default, it returns an empty state, meaning that this report is not implemented.
Users can register a reporting handler when creating the engine. Each connector needs report their status accordingly, which can be returned to the user from this interface in the engine.