Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-4792

Connector stops streaming after a re-balance

    XMLWordPrintable

Details

    • False
    • False

    Description

      i restarted my kafka connect pod and see the below, updated rows in oracle are not flowing into kafka:

       

      i also notice that the jmx metric 'kafka_connect_connector_task_status' has status failed but the kafka connect api (connectors/redact/status) shows running for connector and task status

       

      [2022-02-23 13:20:55,240] INFO Requested thread factory for connector OracleConnector, id = my-topicoracle named = db-history-config-check (io.debezium.util.Threads)
      [2022-02-23 13:20:55,241] ERROR WorkerSourceTask{id=kafka-connectsrc-01-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
      java.lang.RuntimeException: Unable to register the MBean 'debezium.oracle:type=connector-metrics,context=schema-history,server=my-topicoracle'
          at io.debezium.metrics.Metrics.register(Metrics.java:59)
          at io.debezium.relational.history.DatabaseHistoryMetrics.started(DatabaseHistoryMetrics.java:100)
          at io.debezium.relational.history.AbstractDatabaseHistory.start(AbstractDatabaseHistory.java:82)
          at io.debezium.relational.history.KafkaDatabaseHistory.start(KafkaDatabaseHistory.java:249)
          at io.debezium.relational.HistorizedRelationalDatabaseSchema.<init>(HistorizedRelationalDatabaseSchema.java:42)
          at io.debezium.connector.oracle.OracleDatabaseSchema.<init>(OracleDatabaseSchema.java:38)
          at io.debezium.connector.oracle.OracleConnectorTask.start(OracleConnectorTask.java:62)
          at io.debezium.connector.common.BaseSourceTask.start(BaseSourceTask.java:130)
          at org.apache.kafka.connect.runtime.WorkerSourceTask.initializeAndStart(WorkerSourceTask.java:231)
          at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186)
          at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused by: javax.management.InstanceAlreadyExistsException: debezium.oracle:type=connector-metrics,context=schema-history,server=my-topicoracle
          at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436)
          at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855)
          at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955)
          at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890)
          at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
          at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
          at io.debezium.metrics.Metrics.register(Metrics.java:55)
          ... 15 more
      [2022-02-23 13:20:55,241] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask)
      [2022-02-23 13:20:55,249] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-02-23 13:20:55,249] WARN Could not stop task (org.apache.kafka.connect.runtime.WorkerSourceTask)
      java.lang.NullPointerException
          at io.debezium.connector.oracle.OracleConnectorTask.doStop(OracleConnectorTask.java:139)
          at io.debezium.connector.common.BaseSourceTask.stop(BaseSourceTask.java:252)
          at io.debezium.connector.common.BaseSourceTask.stop(BaseSourceTask.java:222)
          at org.apache.kafka.connect.runtime.WorkerSourceTask.close(WorkerSourceTask.java:173)
          at org.apache.kafka.connect.runtime.WorkerTask.doClose(WorkerTask.java:167)
          at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:193)
          at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      [2022-02-23 13:20:55,250] INFO [Producer clientId=connector-producer-kafka-connectsrc-01-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer) 

       

      [2022-02-23 13:20:55,707] INFO Database history topic 'dbz-oracle-history' has correct settings (io.debezium.relational.history.KafkaDatabaseHistory)
      [2022-02-23 13:20:55,707] INFO App info kafka.admin.client for my-topicoracle-dbhistory-topic-check unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:20:55,708] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:55,708] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:55,708] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:56,799] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:56,799] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:56,799] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:56,799] INFO App info kafka.producer for connector-producer-kafka-connectsrc-01-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:20:57,217] INFO [Consumer clientId=my-topicoracle-dbhistory, groupId=my-topicoracle-dbhistory] Cluster ID: x (org.apache.kafka.clients.Metadata)
      [2022-02-23 13:20:57,562] INFO App info kafka.admin.client for connector-adminclient-kafka-connectsrc-01-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:20:57,563] INFO [AdminClient clientId=connector-adminclient-kafka-connectsrc-01-0] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
      org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1645622555059, tries=1, nextAllowedTryMs=-9223372036854774809) timed out at 9223372036854775807 after 1 attempt(s)
      Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call. Call: fetchMetadata
      [2022-02-23 13:20:57,564] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:57,564] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:57,564] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:59,140] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:59,140] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:59,140] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:20:59,141] INFO App info kafka.consumer for my-topicoracle-dbhistory unregistered (org.apache.kafka.common.utils.AppInfoParser) 

       

      [2022-02-23 13:21:06,142] INFO Requested thread factory for connector OracleConnector, id = my-topicoracle named = change-event-source-coordinator (io.debezium.util.Threads)
      [2022-02-23 13:21:06,144] INFO Creating thread debezium-oracleconnector-my-topicoracle-change-event-source-coordinator (io.debezium.util.Threads)
      [2022-02-23 13:21:06,148] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-02-23 13:21:06,149] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} Executing source task (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-02-23 13:21:06,149] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask)
      [2022-02-23 13:21:06,161] INFO Metrics registered (io.debezium.pipeline.ChangeEventSourceCoordinator)
      [2022-02-23 13:21:06,161] INFO Context created (io.debezium.pipeline.ChangeEventSourceCoordinator)
      [2022-02-23 13:21:06,168] INFO The previous offset has been found. (io.debezium.connector.oracle.OracleSnapshotChangeEventSource)
      [2022-02-23 13:21:06,169] INFO Snapshot ended with SnapshotResult [status=SKIPPED, offset=OracleOffsetContext [scn=521737117219]] (io.debezium.pipeline.ChangeEventSourceCoordinator)
      [2022-02-23 13:21:06,172] INFO Connected metrics set to 'false' (io.debezium.pipeline.metrics.DefaultStreamingChangeEventSourceMetrics)
      [2022-02-23 13:21:06,181] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
      [2022-02-23 13:21:06,181] INFO [Producer clientId=my-topicoracle-dbhistory] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
      [2022-02-23 13:21:06,183] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,183] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,183] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,184] INFO App info kafka.producer for my-topicoracle-dbhistory unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:21:06,184] INFO [Producer clientId=connector-producer-kafka-connectsrc-01-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
      [2022-02-23 13:21:06,184] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,184] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,184] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,184] INFO App info kafka.producer for connector-producer-kafka-connectsrc-01-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:21:06,185] INFO App info kafka.admin.client for connector-adminclient-kafka-connectsrc-01-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
      [2022-02-23 13:21:06,186] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,186] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:06,186] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
      [2022-02-23 13:21:15,065] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-02-23 13:21:25,065] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-02-23 13:21:35,066] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask)
      [2022-02-23 13:21:45,066] INFO WorkerSourceTask{id=kafka-connectsrc-01-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) 

      maybe related to https://issues.redhat.com/browse/DBZ-4541 and https://issues.redhat.com/browse/DBZ-1202

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tooptoop toop toop (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: