-
Bug
-
Resolution: Unresolved
-
Major
-
3.3.0.Final
-
None
-
False
-
-
False
I have setup a Debezium connector on a PostgreSQL database with TimescaleDB SMT. It streams data (hypertables) as expected but every day, at the same hour (just after 00:00 UTC), the connector fails (/stop) with a "Failed to read TimescaleDB metadata" error.
A simple connector restart works with something like:
curl -s -XPOST "http://localhost:8083/connectors/my_connector/restart?includeTasks=true&onlyFailed=true"
This seems similar to DBZ-8109 but the issue has been resolved in my Debezium version.
I have many other connectors (without TimescaleDB SMT) running well for years with no similar failure.
I reproduce the same issue in another environment (same database schema but a different data/workload), except that it fails at a different hour (around 4:00pm every day)
What Debezium connector do you use and what version?
- Debian 12.12
- confluent-kafka 7.9.4-1
- Debezium 3.3.0
What is the connector configuration?
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.ApplicationName": "debezium",
"database.dbname": "my_connector",
"database.hostname": "${file:/var/lib/my-secrets/my-connector-debezium:hostname}",
"database.password": "${file:/var/lib/my-secrets/my-connector-debezium:password}",
"database.port": "${file:/var/lib/my-secrets/my-connector-debezium:port}",
"database.ssl.mode": "required",
"database.user": "${file:/var/lib/my-secrets/my-connector-debezium:user}",
"decimal.handling.mode": "double",
"field.name.adjustment.mode": "avro",
"heartbeat.action.query": "update my_debezium_heartbeat set ts=now();",
"heartbeat.interval.ms": "1000",
"incremental.snapshot.chunk.size": "100000",
"name": "my_connector",
"plugin.name": "pgoutput",
"producer.override.compression.type": "lz4",
"publication.name": "debezium_my_connector",
"sanitize.field.names": "true",
"schema.name.adjustment.mode": "avro",
"signal.data.collection": "public.my_debezium_signal",
"slot.name": "debezium_my_connector",
"snapshot.mode": "no_data",
"status.update.interval.ms": "1000",
"table.include.list": "_timescaledb_internal\\._hyper_.+",
"tombstones.on.delete": "false",
"topic.creation.default.cleanup.policy": "delete",
"topic.creation.default.delete.retention.ms": "604800000",
"topic.creation.default.partitions": "20",
"topic.creation.default.replication.factor": "3",
"topic.creation.default.retention.bytes": "-1",
"topic.creation.default.retention.ms": "604800000",
"topic.heartbeat.prefix": "my-ns.debezium.heartbeat",
"topic.prefix": "my-ns.my_connector",
"transforms": "timescaledb,schema,public",
"transforms.public.regex": "my-ns\\.my_connector\\.public\\.(.+)",
"transforms.public.replacement": "my-ns.my_connector.$1",
"transforms.public.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.schema.regex": "my-ns\\.my_connector\\.((?!public).+)\\.(.+)",
"transforms.schema.replacement": "my-ns.my_connector.$1.$2",
"transforms.schema.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.timescaledb.database.dbname": "my_connector",
"transforms.timescaledb.database.hostname": "${file:/var/lib/my-secrets/my-connector-debezium:hostname}",
"transforms.timescaledb.database.password": "${file:/var/lib/my-secrets/my-connector-debezium:password}",
"transforms.timescaledb.database.port": "${file:/var/lib/my-secrets/my-connector-debezium:port}",
"transforms.timescaledb.database.user": "${file:/var/lib/my-secrets/my-connector-debezium:user}",
"transforms.timescaledb.target.topic.prefix": "my-ns.my_connector",
"transforms.timescaledb.type": "io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb"
}
What is the captured database version and mode of deployment?
Database "my_connector" running on PostgreSQL 14.19
On premise (baremetal server) for both Kafka Connect Debezium and PostgreSQL server.
What behavior do you expect?
Connector not to fail every day at the same hour.
What behavior do you see?
Connector fails just after 00:00UTC every day:
[2025-11-14 00:00:25,430] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-15 00:00:39,441] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-16 00:00:31,907] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-17 00:00:33,701] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-18 00:01:27,520] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-19 00:00:31,854] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-20 00:00:49,275] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
[2025-11-21 00:00:37,406] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
Do you see the same behaviour using the latest released Debezium version?
I upgraded from Debezium 2.7.4 to 3.2.3 then 3.3.0 and still have the issue.
I haven't tested version 3.3.1 yet, but it appears that there are no changes to TimescaleDB SMT.
Do you have the connector logs, ideally from start till finish?
Here is the last stack trace:
[2025-11-21 00:00:37,400] INFO WorkerSourceTask{id=my_connector-0} Committing offsets for 10 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2025-11-21 00:00:37,400] INFO WorkerSourceTask{id=my_connector-0} Committing offsets for 1 acknowledged messages (org.apache.kafka.connect.runtime.WorkerSourceTask)
[2025-11-21 00:00:37,406] ERROR WorkerSourceTask{id=my_connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:261)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:180)
at org.apache.kafka.connect.runtime.TransformationChain.apply(TransformationChain.java:57)
at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.sendRecords(AbstractWorkerSourceTask.java:398)
at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.execute(AbstractWorkerSourceTask.java:364)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:226)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:281)
at org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:79)
at org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:238)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.debezium.DebeziumException: Failed to read TimescaleDB metadata
at io.debezium.connector.postgresql.transforms.timescaledb.QueryInformationSchemaMetadata.loadTimescaleMetadata(QueryInformationSchemaMetadata.java:99)
at io.debezium.connector.postgresql.transforms.timescaledb.QueryInformationSchemaMetadata.hypertableId(QueryInformationSchemaMetadata.java:66)
at io.debezium.connector.postgresql.transforms.timescaledb.TimescaleDb.apply(TimescaleDb.java:92)
at org.apache.kafka.connect.runtime.TransformationStage.apply(TransformationStage.java:57)
at org.apache.kafka.connect.runtime.TransformationChain.lambda$apply$0(TransformationChain.java:57)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:208)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:245)
... 13 more
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:399)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:518)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:435)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:357)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:342)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:318)
at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:267)
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:564)
at io.debezium.jdbc.JdbcConnection.query(JdbcConnection.java:506)
at io.debezium.connector.postgresql.transforms.timescaledb.QueryInformationSchemaMetadata.loadTimescaleMetadata(QueryInformationSchemaMetadata.java:83)
... 19 more
Caused by: java.io.EOFException
at org.postgresql.core.PGStream.receiveChar(PGStream.java:479)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2175)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
... 28 more
[2025-11-21 00:00:37,412] INFO Stopping down connector (io.debezium.connector.common.BaseSourceTask)
[2025-11-21 00:00:37,636] INFO Requested thread factory for component JdbcConnection, id = JdbcConnection named = jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,636] ERROR Unexpected exception while performing keepalive status update on the replication stream (io.debezium.connector.postgresql.connection.PostgresReplicationConnection)
java.lang.InterruptedException: sleep interrupted
at java.base/java.lang.Thread.sleep(Native Method)
at io.debezium.util.Metronome$1.pause(Metronome.java:57)
at io.debezium.connector.postgresql.connection.PostgresReplicationConnection$1.lambda$startKeepAlive$0(PostgresReplicationConnection.java:1007)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
[2025-11-21 00:00:37,636] INFO Creating thread debezium-jdbcconnection-JdbcConnection-jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,637] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
[2025-11-21 00:00:37,638] INFO Requested thread factory for component JdbcConnection, id = JdbcConnection named = jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,638] INFO Creating thread debezium-jdbcconnection-JdbcConnection-jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,638] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
[2025-11-21 00:00:37,638] INFO Finished streaming (io.debezium.pipeline.ChangeEventSourceCoordinator)
[2025-11-21 00:00:37,638] INFO Connected metrics set to 'false' (io.debezium.pipeline.ChangeEventSourceCoordinator)
[2025-11-21 00:00:37,639] INFO SignalProcessor stopped (io.debezium.pipeline.signal.SignalProcessor)
[2025-11-21 00:00:37,639] INFO Requested thread factory for component JdbcConnection, id = JdbcConnection named = jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,639] INFO Creating thread debezium-jdbcconnection-JdbcConnection-jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,640] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
[2025-11-21 00:00:37,640] INFO Debezium ServiceRegistry stopped. (io.debezium.service.DefaultServiceRegistry)
[2025-11-21 00:00:37,640] INFO Requested thread factory for component JdbcConnection, id = JdbcConnection named = jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,640] INFO Creating thread debezium-jdbcconnection-JdbcConnection-jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,640] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
[2025-11-21 00:00:37,640] INFO [Producer clientId=connector-producer-my_connector-0] Closing the Kafka producer with timeoutMillis = 30000 ms. (org.apache.kafka.clients.producer.KafkaProducer)
[2025-11-21 00:00:37,642] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,642] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,642] INFO Closing reporter org.apache.kafka.common.telemetry.internals.ClientTelemetryReporter (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,642] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,642] INFO App info kafka.producer for connector-producer-my_connector-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2025-11-21 00:00:37,643] INFO App info kafka.admin.client for connector-adminclient-my_connector-0 unregistered (org.apache.kafka.common.utils.AppInfoParser)
[2025-11-21 00:00:37,643] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,643] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,643] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics)
[2025-11-21 00:00:37,643] INFO Requested thread factory for component JdbcConnection, id = JdbcConnection named = jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,643] INFO Creating thread debezium-jdbcconnection-JdbcConnection-jdbc-connection-close (io.debezium.util.Threads)
[2025-11-21 00:00:37,644] INFO Connection gracefully closed (io.debezium.jdbc.JdbcConnection)
