Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-3343

Debezium stuck in an infinite loop on boot

    XMLWordPrintable

Details

    Description

      Some context on our stack. We have a Kafka Connect container that hosts the Debezium connector, hooking into a Postgres 12 server. We're using pg_output.
       
      Kafka cluster=2.6.0
      dbz-kafka-connect=1.5.0.Beta1
      apicurio=1.3.2.Final
      postgres=v12.4
       
      When starting Debezium our kafka-connect instance continually throws the following logs:
       

      2021-03-22 20:48:56,293 INFO   ||  WorkerSourceTask\{id=booking-manager-ledger-connector-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2021-03-22 20:48:56,293 INFO   ||  WorkerSourceTask\{id=booking-manager-ledger-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2021-03-22 20:48:56,293 WARN   ||  Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart   [io.debezium.connector.common.BaseSourceTask]
      2021-03-22 20:49:56,293 INFO   ||  WorkerSourceTask\{id=booking-manager-ledger-connector-0} Committing offsets   [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2021-03-22 20:49:56,294 INFO   ||  WorkerSourceTask\{id=booking-manager-ledger-connector-0} flushing 0 outstanding messages for offset commit   [org.apache.kafka.connect.runtime.WorkerSourceTask]
      2021-03-22 20:49:56,294 WARN   ||  Couldn't commit processed log positions with the source database due to a concurrent connector shutdown or restart   [io.debezium.connector.common.BaseSourceTask]
      

      Debezium appears to be running the following query to seemingly no end. We've left the system for more than two days with no change in it's behaviour:

       
      SELECT DISTINCT ON (typname) typname, typinput='array_in'::regproc, typtype, sp.r, pg_type.oid FROM pg_catalog.pg_type LEFT JOIN (select ns.oid as nspoid, ns.nspname, r.r from pg_namespace as ns join ( select s.r, (current_schemas(false))[s.r] as nspname from generate_series(1, array_upper(current_schemas(false), 1)) as s(r) ) as r using ( nspname )) as sp ON sp.nspoid = typnamespace ORDER BY typname, sp.r, pg_type.oid
      

      Additionally Debezium refuses to create a replication slot nor publications. There are no events streaming through and no mention of requiring to do a snapshot; the behaviour is the same whether we opt in or out of the snapshot.
       
      We've done a search through the docs and gitter but haven't found any information on resolving this. A user from a while back did mention they were running into the same issue but they did not post a solution https://gitter.im/debezium/user?at=5f0f458be3505d7736004c06. Following the advice of one of the responders https://gitter.im/debezium/user?at=5f0ffcfa49fe2127949ab12e I ran the query manually which completed in ~7 seconds, most of the query listed out a ton of pg_toast_* typname, unsure if that is of any significance?
       
      It was requested that I run a couple of queries and attach their output.
      query2.txt query1_truncated.txt

      Attachments

        1. debezium.log
          26 kB
        2. lock1.txt
          22 kB
        3. lock2.txt
          79 kB
        4. lock3.txt
          79 kB
        5. query1_truncated.txt
          49.97 MB
        6. query2.txt
          23.39 MB
        7. types_result.sql
          263 kB

        Activity

          People

            gmorling@redhat.com Gunnar Morling
            brandonharvey Brandon Harvey (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: