Details
-
Feature Request
-
Resolution: Done
-
Major
-
0.7.0
-
None
-
0
-
0%
Description
After upgrade to version 0.7 history topic is 1.1 GB which is huge
No matter how much we will increase the
database.history.kafka.recovery.attempts
database.history.kafka.recovery.poll.interval.ms
we get this error
The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms
We are getting this error below.
I have to mention i am using only 33 table and my history is 2 weeks old and my topic is huge .1.1 GB
I think it will be issue for everyone.
I think we should add the history only the white-list (33) not all.
This is not a enhancement rather a blocker bug since the only bypass u have is to delete the topic and start from blank each time it's happen.
the full error:
"2018-01-14 13:54:51,249] ERROR WorkerSourceTask{id=ks-connector4032-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask:172)
org.apache.kafka.connect.errors.ConnectException: java.lang.IllegalStateException: The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:194)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:157)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: The database history couldn't be recovered. Consider to increase the value for database.history.kafka.recovery.poll.interval.ms
at io.debezium.relational.history.KafkaDatabaseHistory.recoverRecords(KafkaDatabaseHistory.java:196)
at io.debezium.relational.history.AbstractDatabaseHistory.recover(AbstractDatabaseHistory.java:61)
at io.debezium.connector.mysql.MySqlSchema.loadHistory(MySqlSchema.java:249)
at io.debezium.connector.mysql.MySqlTaskContext.loadHistory(MySqlTaskContext.java:122)
at io.debezium.connector.mysql.MySqlConnectorTask.start(MySqlConnectorTask.java:80)
... 8 more
"
