-
Bug
-
Resolution: Obsolete
-
Critical
-
None
-
None
-
None
-
True
-
-
False
We suspect this cause of the issue. The TCP connection remains ESTABLISHED, but the MySQL session is DEAD, leaving the connection in a stale state indefinitely.
In this scenario, the connector still shows as RUNNING, which is the problem.
Possible causes: * MySQL process termination
- Network layer issues
- MySQL server-side timeouts:
- Connection limits reached
- Idle connection timeout (MySQL closes inactive sessions)
Debezium Connector → TCP Layer → MySQL Application Layer
1. TCP Handshake (Layer 4)
├── SYN packet sent
├── SYN-ACK received
└── ACK sent → TCP ESTABLISHED
2. MySQL Authentication (Layer 7)
├── MySQL handshake packet
├── Username/password exchange
└── Authentication OK → MySQL Session Active
3. Binlog Streaming (Layer 7)
├── SHOW MASTER STATUS
├── COM_BINLOG_DUMP command
└── Binlog events flowing → CDC Working
One of the Problem Scenario:Debezium Connector
↓
TCP Connection: ESTABLISHED
← Shows "Connected"
↓
MySQL Session: DEAD
← Actually disconnected
↓
Result: Connector thinks it's working but no data flows
- TCP (Layer 4) only knows about network connectivity
- MySQL (Layer 7) handles actual database sessions
Replicating the Issue
To reproduce the issue, I am running a MySQL database in a local Docker container and have completed the local setup to run a MySQL Debezium source Kafka connector (version 3.3.2.Final).
I tested the end-to-end flow successfully. When I make changes to a database table, the correct events are pushed to the Kafka topic. I have also configured a heartbeat with a 30-second interval, and the heartbeat topic is receiving heartbeat events as expected. Also i configured the heartbeat.action.query to insert current timestamp.
When I pause the MySQL Docker container, the connector does not fail. Instead, the keepalive thread continuously retries the database connection, which can be observed in the Kafka Connect cluster logs. However, the connector task itself does not log any errors or exceptions. Also heartbeat events is stopped.
When I unpause the MySQL Docker container, the keepalive thread restores the connection successfully. This reconnection works correctly in my local PC environment.
However, when the database and the connector are running on different servers, the connection is not restored successfully even when the db is active and able to make new connection. In this scenario, when we restart the connector, it re-establishes the connection and events start flowing again.
The expectation is that, at the connector level, there should be a mechanism to re-establish the connection using the keepalive thread, automatically restart the connector, retry the connection, or at least fail the connector explicitly.
- clones
-
DBZ-9247 Throw an exception on missing heartbeat table on Debezium-connector-postgres
-
- Closed
-