-
Task
-
Resolution: Done
-
Major
-
None
-
None
-
False
-
None
-
False
-
-
KIP-898, introduced in Kafka 3.6, changes to the way plugins are loaded in Connect.
Now all plugins can be loaded using service loader manifests. In fact the plan is to make this the default mechanism to load plugins in the future.
Since Kafka 3.6, Connect users can opt-in this feature by setting the plugin.discovery configuration.
Debezium should add service loader manifests for all connectors, transformations, converters and predicates so it's usable when plugin.discovery is set to service_load or hybrid_fail.
For example, Connect with plugin.discovery=service_load does not detect the Debezium plugins, and with plugin.discovery=hybrid_fail it errors at startup:
ERROR Stopping due to error (org.apache.kafka.connect.cli.AbstractConnectCli:100) org.apache.kafka.connect.errors.ConnectException: One or more plugins are missing ServiceLoader manifests may not be usable with plugin.discovery=service_load: [ file:/tmp/plugins/debezium-connector-mysql/ io.debezium.connector.mysql.MySqlConnector source 2.5.0.Final file:/tmp/plugins/debezium-connector-mysql/ io.debezium.connector.mysql.transforms.ReadToInsertEvent transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.converters.BinaryDataConverter converter undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.converters.BinaryDataConverter header_converter undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.converters.ByteArrayConverter converter undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.converters.ByteArrayConverter header_converter undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.converters.CloudEventsConverter converter undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.ByLogicalTableRouter transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.ExtractChangedRecordState transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.ExtractNewRecordState transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.ExtractSchemaToNewRecord transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.HeaderToValue transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.SchemaChangeEventFilter transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.TimezoneConverter transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.outbox.EventRouter transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.partitions.PartitionRouting transformation undefined file:/tmp/plugins/debezium-connector-mysql/ io.debezium.transforms.tracing.ActivateTracingSpan transformation undefined ] Read the documentation at https://kafka.apache.org/documentation.html#connect_plugindiscovery for instructions on migrating your plugins to take advantage of the performance improvements of service_load mode. To silence this error, set plugin.discovery=hybrid_warn in the worker config. at org.apache.kafka.connect.runtime.isolation.Plugins.maybeReportHybridDiscoveryIssue(Plugins.java:124) at org.apache.kafka.connect.runtime.isolation.Plugins.initLoaders(Plugins.java:92) at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:74) at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:64) at org.apache.kafka.connect.cli.AbstractConnectCli.startConnect(AbstractConnectCli.java:121) at org.apache.kafka.connect.cli.AbstractConnectCli.run(AbstractConnectCli.java:94) at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:116)