Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-6510

Utilize event.processing.failure.handling.mode in Vitess replication connection

XMLWordPrintable

    • Icon: Feature Request Feature Request
    • Resolution: Done
    • Icon: Major Major
    • 2.3.0.CR1
    • None
    • vitess-connector
    • None

      Feature request or enhancement

      For feature requests or enhancements, provide this information, please:

      Which use case/requirement will be addressed by the proposed feature?

      In some cases, there are errors when the events are being consumed from the vstream. We should use the property event.processing.failure.handling.mode for handling events properly. This is similar to how it's done for postgres.

      One example of this we observed in our deploy of Debezium is when the message exceeds the max grpc size. Users may often want to skip these oversized messages since Kafka cannot handle them efficiently. Currently, enabling skip will still cause the connector to fail with this exception:

      io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: gRPC message exceeds maximum size 4194304: 6653790        at io.grpc.Status.asRuntimeException(Status.java:533)        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)        at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)        at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427)        at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460)        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:616)        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:802)        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:781)        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)        at java.base/java.lang.Thread.run(Thread.java:829)

      Implementation ideas (optional)

      Add handling to the onError method of the VitessReplicationConnection to allow for reading this config property and handling errors appropriately

              Unassigned Unassigned
              tthorn Thomas Thornton
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: