Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-2118

Transaction opened by Debezium is left idle and never committed

    XMLWordPrintable

Details

    • Hide

      Not sure. But it seems like a change to an unrelated type or table that is not whitelisted triggers this runtime query from Debezium to Postgres.

      Show
      Not sure. But it seems like a change to an unrelated type or table that is not whitelisted triggers this runtime query from Debezium to Postgres.
    • Hide

      Delete idle transaction

      Show
      Delete idle transaction

    Description

      A day or so after running the Postgres connector we notice that there is a query that is idle in transaction. After updating to 1.2.0Beta2 this is the latest transaction that is idle:

      SELECT t.oid AS oid, t.typname AS name, t.typelem AS element, t.typbasetype AS parentoid, t.typtypmod as modifiers, t.typcategory as category, e.values as enum_values FROM pg_catalog.pg_type t JOIN pg_catalog.pg_namespace n ON (t.typnamespace = n.oid) LEFT JOIN (SELECT t.enumtypid as id, array_agg(t.enumlabel) as values FROM pg_catalog.pg_enum t GROUP BY id) e ON (t.oid = e.id) WHERE n.nspname != 'pg_toast' AND t.oid = $1
      

      This query seems to be triggered during steady state operation when an unknown SQL type is encountered: https://github.com/debezium/debezium/blob/master/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/TypeRegistry.java#L66-L70

      We have looked into our application and don't see any relevant logs however one time we did notice an NPE that was caught and logged in an older version 1.0.2Final

      01:04:44.047 [debezium-postgresconnector-overwatch-change-event-source-coordinator-] WARN  io.debezium.connector.postgresql.TypeRegistry - Failed to obtain SQL type information for type _qrtz_job_details via custom statement, falling back to TypeInfo#getSQLType() 
      java.lang.NullPointerException: null
          at io.debezium.connector.postgresql.TypeRegistry$SqlTypeMapper.getSqlType(TypeRegistry.java:559) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
          at io.debezium.connector.postgresql.TypeRegistry.prime(TypeRegistry.java:371) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
          at io.debezium.connector.postgresql.TypeRegistry.<init>(TypeRegistry.java:122) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
          at io.debezium.connector.postgresql.connection.PostgresConnection.<init>(PostgresConnection.java:72) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
          at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.handleRelationMessage(PgOutputMessageDecoder.java:268) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
          at io.debezium.connector.postgresql.connection.pgoutput.PgOutputMessageDecoder.processMessage(PgOutputMessageDecoder.java:189) [debezium-connector-postgres-1.0.2.Final.jar:1.0.2.Final]
      

      The idle transaction prevents our database from running automatic vacuum jobs and CPU steadily increases until it is overloaded. After deleting the idle transaction everything returns to normal.

      Attachments

        Activity

          People

            jpechane Jiri Pechanec
            sovietaced Jason Parraga (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: