Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2089

Add API for not registering metrics MBean into the platform MBean server

    • Icon: Enhancement Enhancement
    • Resolution: Done
    • Icon: Major Major
    • 1.2.0.CR1
    • 1.1.0.Final
    • core-library
    • None

      Registration of MBeans
      (see https://github.com/debezium/debezium/blob/master/debezium-core/src/main/java/io/debezium/metrics/Metrics.java#L45 and https://github.com/debezium/debezium/blob/1.1/debezium-core/src/main/java/io/debezium/pipeline/ChangeEventSourceCoordinator.java#L80)
      can not be skipped by any API.

      In native mode in GRaalMV, there is no JMX.

      It would be very helpful for camel-quarkus and its debezium extensions, to have an option to skip MBean registration.

            [DBZ-2089] Add API for not registering metrics MBean into the platform MBean server

            Released

            Jiri Pechanec added a comment - Released

            Gunnar Morling added a comment - - edited

            Thanks for the quick feedback, jondruse@redhat.com. I suppose this works with MySQL because that driver already has been "quarkified". Might be different when it comes to other DBZ connectors.

            Gunnar Morling added a comment - - edited Thanks for the quick feedback, jondruse@redhat.com . I suppose this works with MySQL because that driver already has been "quarkified". Might be different when it comes to other DBZ connectors.

            jpechane I've tested it and fix works correctly.

            Jiri Ondrusek added a comment - jpechane I've tested it and fix works correctly.

            Jiri Pechanec added a comment - - edited

            jondruse@redhat.com COuld you please try to test the linked PR?

            Jiri Pechanec added a comment - - edited jondruse@redhat.com COuld you please try to test the linked PR?

            Hi jpechane,
            I've created extension for debeziun-postgres into camel-quarkus (https://github.com/apache/camel-quarkus/tree/master/extensions/debezium-postgres)
            In case the metrics is registering in native, I'm getting following exception, which indicates that mbean is null:

            020-05-21 11:55:33,112 ERROR [io.deb.emb.EmbeddedEngine] (Camel (camel-1) thread #0 - DebeziumConsumer) Unable to initialize and start connector's task class 'io.debezium.connector.mysql.MySqlConnectorTask' with config:

            Unknown macro: {snapshot.locking.mode=minimal, connector.class=io.debezium.connector.mysql.MySqlConnector, include.schema.changes=true, database.jdbc.driver=com.mysql.cj.jdbc.Driver, poll.interval.ms=500, database.history.kafka.recovery.poll.interval.ms=100, heartbeat.topics.prefix=__debezium-heartbeat, binlog.buffer.size=0, snapshot.fetch.size=0, offset.commit.policy=io.debezium.embedded.spi.OffsetCommitPolicy$PeriodicCommitOffsetPolicy, database.user=root, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, heartbeat.interval.ms=0, source.struct.version=v2, inconsistent.schema.handling.mode=fail, enable.time.adjuster=true, gtid.new.channel.position=earliest, database.password=********, internal.value.converter=org.apache.kafka.connect.json.JsonConverter, name=localhost, gtid.source.filter.dml.events=true, database.history.store.only.monitored.tables.ddl=false, max.batch.size=2048, connect.keep.alive=true, database.history=io.debezium.relational.history.FileDatabaseHistory, snapshot.mode=initial, connect.timeout.ms=30000, max.queue.size=8192, snapshot.delay.ms=0, database.history.kafka.recovery.attempts=100, tombstones.on.delete=false, offset.storage.file.filename=/tmp/debezium-mysql-store-4858804405295680308, decimal.handling.mode=precise, snapshot.new.tables=off, offset.storage.partitions=0, database.history.skip.unparseable.ddl=false, table.ignore.builtin=true, database.history.file.filename=/tmp/debezium-mysql-history-file-3912440284381919565, bigint.unsigned.handling.mode=long, database.server.id=223344, event.deserialization.failure.handling.mode=fail, time.precision.mode=adaptive_time_microseconds, offset.flush.timeout.ms=5000, database.server.name=qa, event.processing.failure.handling.mode=fail, database.port=32837, offset.flush.interval.ms=60000, internal.key.converter=org.apache.kafka.connect.json.JsonConverter, database.ssl.mode=disabled, database.hostname=localhost, database.server.id.offset=10000, offset.storage.replication.factor=0, connect.keep.alive.interval.ms=60000, include.query=false}

            : org.apache.kafka.connect.errors.ConnectException: java.lang.NullPointerException
            at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:289)
            at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:104)
            at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:722)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.lang.Thread.run(Thread.java:834)
            at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:497)
            at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
            Caused by: java.lang.NullPointerException
            at io.debezium.metrics.Metrics.register(Metrics.java:47)
            at io.debezium.relational.history.DatabaseHistoryMetrics.started(DatabaseHistoryMetrics.java:100)
            at io.debezium.relational.history.AbstractDatabaseHistory.start(AbstractDatabaseHistory.java:58)
            at io.debezium.relational.history.FileDatabaseHistory.start(FileDatabaseHistory.java:68)
            at io.debezium.connector.mysql.MySqlSchema.start(MySqlSchema.java:162)
            at io.debezium.connector.mysql.MySqlTaskContext.start(MySqlTaskContext.java:246)
            at io.debezium.connector.mysql.MySqlConnectorTask.createAndStartTaskContext(MySqlConnectorTask.java:349)
            at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:143)
            ... 9 more

            Jiri Ondrusek added a comment - Hi jpechane , I've created extension for debeziun-postgres into camel-quarkus ( https://github.com/apache/camel-quarkus/tree/master/extensions/debezium-postgres ) In case the metrics is registering in native, I'm getting following exception, which indicates that mbean is null: 020-05-21 11:55:33,112 ERROR [io.deb.emb.EmbeddedEngine] (Camel (camel-1) thread #0 - DebeziumConsumer) Unable to initialize and start connector's task class 'io.debezium.connector.mysql.MySqlConnectorTask' with config: Unknown macro: {snapshot.locking.mode=minimal, connector.class=io.debezium.connector.mysql.MySqlConnector, include.schema.changes=true, database.jdbc.driver=com.mysql.cj.jdbc.Driver, poll.interval.ms=500, database.history.kafka.recovery.poll.interval.ms=100, heartbeat.topics.prefix=__debezium-heartbeat, binlog.buffer.size=0, snapshot.fetch.size=0, offset.commit.policy=io.debezium.embedded.spi.OffsetCommitPolicy$PeriodicCommitOffsetPolicy, database.user=root, offset.storage=org.apache.kafka.connect.storage.FileOffsetBackingStore, heartbeat.interval.ms=0, source.struct.version=v2, inconsistent.schema.handling.mode=fail, enable.time.adjuster=true, gtid.new.channel.position=earliest, database.password=********, internal.value.converter=org.apache.kafka.connect.json.JsonConverter, name=localhost, gtid.source.filter.dml.events=true, database.history.store.only.monitored.tables.ddl=false, max.batch.size=2048, connect.keep.alive=true, database.history=io.debezium.relational.history.FileDatabaseHistory, snapshot.mode=initial, connect.timeout.ms=30000, max.queue.size=8192, snapshot.delay.ms=0, database.history.kafka.recovery.attempts=100, tombstones.on.delete=false, offset.storage.file.filename=/tmp/debezium-mysql-store-4858804405295680308, decimal.handling.mode=precise, snapshot.new.tables=off, offset.storage.partitions=0, database.history.skip.unparseable.ddl=false, table.ignore.builtin=true, database.history.file.filename=/tmp/debezium-mysql-history-file-3912440284381919565, bigint.unsigned.handling.mode=long, database.server.id=223344, event.deserialization.failure.handling.mode=fail, time.precision.mode=adaptive_time_microseconds, offset.flush.timeout.ms=5000, database.server.name=qa, event.processing.failure.handling.mode=fail, database.port=32837, offset.flush.interval.ms=60000, internal.key.converter=org.apache.kafka.connect.json.JsonConverter, database.ssl.mode=disabled, database.hostname=localhost, database.server.id.offset=10000, offset.storage.replication.factor=0, connect.keep.alive.interval.ms=60000, include.query=false} : org.apache.kafka.connect.errors.ConnectException: java.lang.NullPointerException at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:289) at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:104) at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:722) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:497) at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193) Caused by: java.lang.NullPointerException at io.debezium.metrics.Metrics.register(Metrics.java:47) at io.debezium.relational.history.DatabaseHistoryMetrics.started(DatabaseHistoryMetrics.java:100) at io.debezium.relational.history.AbstractDatabaseHistory.start(AbstractDatabaseHistory.java:58) at io.debezium.relational.history.FileDatabaseHistory.start(FileDatabaseHistory.java:68) at io.debezium.connector.mysql.MySqlSchema.start(MySqlSchema.java:162) at io.debezium.connector.mysql.MySqlTaskContext.start(MySqlTaskContext.java:246) at io.debezium.connector.mysql.MySqlConnectorTask.createAndStartTaskContext(MySqlConnectorTask.java:349) at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:143) ... 9 more

            jondruse@redhat.com Hi, what is the behviour, is there any exception thrown? We might solve this in most easiest way by detecting such situation and skipping the registration. Is for exmaple platform mbean server null?

            Jiri Pechanec added a comment - jondruse@redhat.com Hi, what is the behviour, is there any exception thrown? We might solve this in most easiest way by detecting such situation and skipping the registration. Is for exmaple platform mbean server null?

            Ok, wow, we didn't test yet at all using Debezium connectors natively with GraalVM I'd expect the MBeans not to be the only problem, but also DB drivers themselves. This is to say we certainly can make it so JMX can be turned off, but it probably won't be the only thing needed to enable Debezium on GraalVM. We'd also have to look into exporting metrics via different means (MicroProfile Metrics) as an alternative way for monitoring the connectors in this case.

            Gunnar Morling added a comment - Ok, wow, we didn't test yet at all using Debezium connectors natively with GraalVM I'd expect the MBeans not to be the only problem, but also DB drivers themselves. This is to say we certainly can make it so JMX can be turned off, but it probably won't be the only thing needed to enable Debezium on GraalVM. We'd also have to look into exporting metrics via different means (MicroProfile Metrics) as an alternative way for monitoring the connectors in this case.

              jpechane Jiri Pechanec
              jondruse@redhat.com Jiri Ondrusek
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: