Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: under-triaging
Affects Version/s: 2.7.1.Final, 3.0.0.Beta1
Component/s: vitess-connector
Labels:
None

Blocked:
False
Blocked Reason:
None
Ready:
False

Severity:
Important

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

What Debezium connector do you use and what version?

Debezium vitess connector 2.7.1 Final (also tried with 3.0.0.Beta1)

What is the connector configuration?

apiVersion: kafka.strimzi.io/v1beta2
    kind: KafkaConnector
    metadata:
      name: debezium-connector-betonred-cx
      labels:
        strimzi.io/cluster: debezium-connect-cluster
    spec:
      class: io.debezium.connector.vitess.VitessConnector
      tasksMax: 1
      autoRestart:
        enabled: true
        maxRestarts: 2
      state: running
      config:
        errors.max.retries: 2
        schema.history.internal.consumer.sasl.mechanism: SCRAM-SHA-512
        schema.history.internal.consumer.security.protocol: SASL_PLAINTEXT
        schema.history.internal.producer.sasl.mechanism: SCRAM-SHA-512
        schema.history.internal.producer.security.protocol: SASL_PLAINTEXT
        max.request.size: 52000000
        producer.max.request.size: 52000000
        database.hostname: *****.net
        database.port: 15992
        database.server.id: 155120
        database.connectionTimeZone: Europe/Moscow
        vitess.keyspace: betonred_cx
        vitess.tablet.type: REPLICA
        vitess.vtctld.host: vitess103.marfa-tech.net
        vitess.vtctld.port: 15999
        # topics settings
        topic.prefix: betonred_cx
        table.exclude.list: betonred_cx\.\S*[^A-Za-z0-9]\S*
        schema.history.internal.kafka.bootstrap.servers: kafka-cluster-kafka-bootstrap:9092
        schema.history.internal.kafka.topic: schema-changes.betonred_cx
        provide.transaction.metadata: true
        key.converter: io.confluent.connect.avro.AvroConverter
        key.converter.schema.registry.url: http://schema-registry:8081
        value.converter: io.confluent.connect.avro.AvroConverter
        value.converter.schema.registry.url: http://schema-registry:8081
        snapshot.mode: never
        tombstones.on.delete: false

What is the captured database version and mode of deployment?

Vitess 17.06

What behavior do you expect?

Correct work of connector

What behavior do you see?

Connector failed with errors

Do you see the same behaviour using the latest released Debezium version?

Yes

Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide DEBUG/TRACE level log)

2024-08-30 12:03:38,751 ERROR [debezium-connector-betonred-cx|task-0] Producer failure (io.debezium.pipeline.ErrorHandler) [debezium-vitessconnector-betonred_cx-change-event-source-coordinator]
java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype
  at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)
  at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
  at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
  at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
  at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)
  at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:101)
  at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)
  at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.newChannel(VitessReplicationConnection.java:338)
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.startStreaming(VitessReplicationConnection.java:83)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:67)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)

2024-08-30 12:03:38,751 ERROR [debezium-connector-betonred-cx|task-0] Failed to close replicationConnection (io.debezium.connector.vitess.VitessStreamingChangeEventSource) [debezium-vitessconnector-betonred_cx-change-event-source-coordinator]
java.lang.NullPointerException: Cannot invoke "io.grpc.ManagedChannel.shutdownNow()" because the return value of "java.util.concurrent.atomic.AtomicReference.get()" is null
  at io.debezium.connector.vitess.connection.VitessReplicationConnection.close(VitessReplicationConnection.java:350)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:84)
  at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)
  at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)
  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  at java.base/java.lang.Thread.run(Thread.java:840)

2024-08-30 12:03:39,250 ERROR [debezium-connector-betonred-cx|task-0] WorkerSourceTask{id=debezium-connector-betonred-cx-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-debezium-connector-betonred-cx-0] org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.   at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:67)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:79)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:30)   at io.debezium.pipeline.ChangeEventSourceCoordinator.streamEvents(ChangeEventSourceCoordinator.java:312)   at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:203)   at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:143)   at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)   at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.util.ServiceConfigurationError: io.grpc.ManagedChannelProvider: io.grpc.netty.shaded.io.grpc.netty.NettyChannelProvider not a subtype   at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1244)   at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)   at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)   at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)   at io.grpc.ServiceProviders.loadAll(ServiceProviders.java:67)   at io.grpc.ManagedChannelRegistry.getDefaultRegistry(ManagedChannelRegistry.java:101)   at io.grpc.ManagedChannelProvider.provider(ManagedChannelProvider.java:43)   at io.grpc.ManagedChannelBuilder.forAddress(ManagedChannelBuilder.java:39)   at io.debezium.connector.vitess.connection.VitessReplicationConnection.newChannel(VitessReplicationConnection.java:338)   at io.debezium.connector.vitess.connection.VitessReplicationConnection.startStreaming(VitessReplicationConnection.java:83)   at io.debezium.connector.vitess.VitessStreamingChangeEventSource.execute(VitessStreamingChangeEventSource.java:67)   ... 9 more

How to reproduce the issue using our tutorial deployment?

Use this Dockerfile for build docker image

FROM alpine as downloader

ENV DEBEZIUM_CONNECTOR_MYSQL_VERSION='2.7.0.Final' \
    DEBEZIUM_CONNECTOR_POSTGRES_VERSION='2.7.0.Final' \
    DEBEZIUM_CONNECTOR_VITESS_VERSION='2.7.0.Final' \
    CONFLUENT_AVRO_VERSION='7.6.2' \
    APACHE_AVRO='1.11.3'

RUN mkdir -p /plugins/confluent
WORKDIR /plugins
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-mysql/$DEBEZIUM_CONNECTOR_MYSQL_VERSION/debezium-connector-mysql-$DEBEZIUM_CONNECTOR_MYSQL_VERSION-plugin.tar.gz
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/$DEBEZIUM_CONNECTOR_POSTGRES_VERSION/debezium-connector-postgres-$DEBEZIUM_CONNECTOR_POSTGRES_VERSION-plugin.tar.gz
RUN wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-vitess/$DEBEZIUM_CONNECTOR_VITESS_VERSION/debezium-connector-vitess-$DEBEZIUM_CONNECTOR_VITESS_VERSION-plugin.tar.gz

RUN wget -P /plugins/confluent https://repo1.maven.org/maven2/org/apache/avro/avro/$APACHE_AVRO/avro-$APACHE_AVRO.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/common-config/$CONFLUENT_AVRO_VERSION/common-config-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/common-utils/$CONFLUENT_AVRO_VERSION/common-utils-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-connect-avro-converter/$CONFLUENT_AVRO_VERSION/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-connect-avro-data/$CONFLUENT_AVRO_VERSION/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-avro-serializer/$CONFLUENT_AVRO_VERSION/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-serializer/$CONFLUENT_AVRO_VERSION/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-registry-client/$CONFLUENT_AVRO_VERSION/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar
RUN wget -P /plugins/confluent https://packages.confluent.io/maven/io/confluent/kafka-schema-converter/$CONFLUENT_AVRO_VERSION/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar

RUN for f in *.tar.gz; do tar -zxvf "$f"; done

# Image used by strimzi operator for your kafka cluser
FROM quay.io/strimzi/kafka:0.42.0-kafka-3.7.1
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
RUN mkdir -p /opt/kafka/plugins/confluent
ENV CONFLUENT_AVRO_VERSION='7.6.2'
ENV APACHE_AVRO='1.11.3'

COPY --from=downloader /plugins/debezium-connector-mysql/* /opt/kafka/plugins/debezium-connector-mysql/
COPY --from=downloader /plugins/debezium-connector-postgres/* /opt/kafka/plugins/debezium-connector-postgres/
COPY --from=downloader /plugins/debezium-connector-vitess/* /opt/kafka/plugins/debezium-connector-vitess/
COPY --from=downloader /plugins/confluent/common-config-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/common-config-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/common-utils-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/common-utils-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-connect-avro-converter-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-connect-avro-data-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-avro-serializer-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-serializer-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-registry-client-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar /opt/kafka/plugins/confluent/kafka-schema-converter-$CONFLUENT_AVRO_VERSION.jar
COPY --from=downloader /plugins/confluent/avro-$APACHE_AVRO.jar /opt/kafka/plugins/confluent/avro-$APACHE_AVRO.jar


USER 1001

Details

Description

What Debezium connector do you use and what version?

What is the connector configuration?

What is the captured database version and mode of deployment?

What behavior do you expect?

What behavior do you see?

Do you see the same behaviour using the latest released Debezium version?

Do you have the connector logs, ideally from start till finish?

How to reproduce the issue using our tutorial deployment?

Attachments

Easy Agile Planning Poker

Activity

People

Dates