Uploaded image for project: 'Debezium'
  1. Debezium
  2. DBZ-7701

RedisSchemaHistoryIT#testRedisConnectionRetry can run into infinite retry loop

XMLWordPrintable

      RedisSchemaHistoryIT#testRedisConnectionRetry can run into infinite loop when the connector is paused, data are stored into mysql but for some reason test doesn't continue and doesn't unpause the Redis container so Debezium server running in another thread loops in infinite retry until it's killed by GH after several hours.

      024-03-22T00:31:14.5106807Z 2024-03-22 00:31:14,498 INFO  [io.deb.con.mys.MySqlSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Reading structure of database 'inventory'
      2024-03-22T00:31:14.7109658Z 2024-03-22 00:31:14,678 INFO  [io.deb.rel.RelationalSnapshotChangeEventSource] (debezium-mysqlconnector-testc-change-event-source-coordinator) Snapshot step 6 - Persisting schema history
      2024-03-22T00:31:14.7111976Z Pausing container
      2024-03-22T00:31:14.8110636Z 2024-03-22 00:31:14,762 INFO  [io.deb.con.CommonConnectorConfig] (main) Loading the custom source info struct maker plugin: io.debezium.connector.mysql.MySqlSourceInfoStructMaker
      2024-03-22T00:31:14.8113408Z 2024-03-22 00:31:14,780 INFO  [io.deb.con.mys.MySqlConnectorConfig] (main) Using io.debezium.connector.mysql.strategy.mysql.MySqlConnectorAdapter
      2024-03-22T00:31:15.3114403Z Creating new redis_test table and inserting 5 records to it
      2024-03-22T00:31:16.8131519Z 2024-03-22 00:31:16,724 WARN  [io.deb.sto.red.his.RedisSchemaHistory] (debezium-mysqlconnector-testc-change-event-source-coordinator) Connection to Redis failed, will try to reconnect [attempt 1 of 10]
      2024-03-22T00:31:17.1135663Z 2024-03-22 00:31:17,068 ERROR [io.deb.ser.red.RedisStreamChangeConsumer] (pool-7-thread-1) Connection error: io.debezium.storage.redis.RedisClientConnectionException: redis.clients.jedis.exceptions.JedisConnectionException: 
      java.net.SocketTimeoutException: Read timed out
      2024-03-22T00:31:17.1137848Z    at io.debezium.storage.redis.JedisClient.tryErrors(JedisClient.java:119)
      2024-03-22T00:31:17.1138847Z    at io.debezium.storage.redis.JedisClient.info(JedisClient.java:125)
      2024-03-22T00:31:17.1139837Z    at io.debezium.server.redis.RedisMemoryThreshold.memoryTuple(RedisMemoryThreshold.java:105)
      2024-03-22T00:31:17.1141013Z    at io.debezium.server.redis.RedisMemoryThreshold.checkMemory(RedisMemoryThreshold.java:58)
      2024-03-22T00:31:17.1142248Z    at io.debezium.server.redis.RedisStreamChangeConsumer.lambda$handleBatch$3(RedisStreamChangeConsumer.java:184)
      2024-03-22T00:31:17.1143222Z    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
      2024-03-22T00:31:17.1143979Z    at java.base/java.util.stream.IntPipeline$1$1.accept(IntPipeline.java:180)
      2024-03-22T00:31:17.1144790Z    at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104)
      2024-03-22T00:31:17.1145903Z    at java.base/java.util.Spliterator$OfInt.forEachRemaining(Spliterator.java:711)
      2024-03-22T00:31:17.1146728Z    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
      2024-03-22T00:31:17.1147600Z    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
      2024-03-22T00:31:17.1148504Z    at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
      2024-03-22T00:31:17.1149584Z    at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
      2024-03-22T00:31:17.1150459Z    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
      2024-03-22T00:31:17.1151283Z    at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
      2024-03-22T00:31:17.1152231Z    at io.debezium.server.redis.RedisStreamChangeConsumer.handleBatch(RedisStreamChangeConsumer.java:147)
      2024-03-22T00:31:17.1153421Z    at io.debezium.embedded.async.ParallelSmtAndConvertBatchProcessor.processRecords(ParallelSmtAndConvertBatchProcessor.java:56)
      2024-03-22T00:31:17.1154508Z    at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1128)
      2024-03-22T00:31:17.1155415Z    at io.debezium.embedded.async.AsyncEmbeddedEngine$PollRecords.doCall(AsyncEmbeddedEngine.java:1109)
      2024-03-22T00:31:17.1156220Z    at io.debezium.embedded.async.RetryingCallable.call(RetryingCallable.java:47)
      2024-03-22T00:31:17.1156897Z    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      2024-03-22T00:31:17.1157573Z    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
      2024-03-22T00:31:17.1158251Z    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      2024-03-22T00:31:17.1158988Z    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
      2024-03-22T00:31:17.1159829Z    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
      2024-03-22T00:31:17.1160711Z    at java.base/java.lang.Thread.run(Thread.java:840)
      2024-03-22T00:31:17.1161477Z Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out
      2024-03-22T00:31:17.1162395Z    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:204)
      2024-03-22T00:31:17.1163151Z    at redis.clients.jedis.util.RedisInputStream.readByte(RedisInputStream.java:42)
      2024-03-22T00:31:17.1163802Z    at redis.clients.jedis.Protocol.process(Protocol.java:126)
      2024-03-22T00:31:17.1164324Z    at redis.clients.jedis.Protocol.read(Protocol.java:192)
      2024-03-22T00:31:17.1164993Z    at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:316)
      2024-03-22T00:31:17.1165752Z    at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:262)
      2024-03-22T00:31:17.1166412Z    at redis.clients.jedis.Connection.getBulkReply(Connection.java:252)
      2024-03-22T00:31:17.1166952Z    at redis.clients.jedis.Jedis.info(Jedis.java:3314)
      2024-03-22T00:31:17.1167531Z    at io.debezium.storage.redis.JedisClient.lambda$info$13(JedisClient.java:125)
      2024-03-22T00:31:17.1168212Z    at io.debezium.storage.redis.JedisClient.tryErrors(JedisClient.java:116)
      2024-03-22T00:31:17.1168695Z    ... 25 more
      2024-03-22T00:31:17.1169039Z Caused by: java.net.SocketTimeoutException: Read timed out
      2024-03-22T00:31:17.1169626Z    at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:288)
      2024-03-22T00:31:17.1170267Z    at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)
      2024-03-22T00:31:17.1170870Z    at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355)
      2024-03-22T00:31:17.1171457Z    at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808)
      2024-03-22T00:31:17.1172056Z    at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966)
      2024-03-22T00:31:17.1172619Z    at java.base/java.io.InputStream.read(InputStream.java:218)
      2024-03-22T00:31:17.1173269Z    at redis.clients.jedis.util.RedisInputStream.ensureFill(RedisInputStream.java:198)
      2024-03-22T00:31:17.1173806Z    ... 34 more
      2024-03-22T00:31:17.1173942Z 
      2024-03-22T00:31:17.4138393Z 2024-03-22 00:31:17,378 INFO  [io.deb.sto.red.RedisCommonConfig] (pool-7-thread-1) Configuration for 'RedisStreamChangeConsumerConfig' with prefix 'debezium.sink.': {type=redis, redis.address=localhost:16379}
      2024-03-22T00:31:19.1153927Z 2024-03-22 00:31:19,028 WARN  [io.deb.sto.red.his.RedisSchemaHistory] (debezium-mysqlconnector-testc-change-event-source-coordinator) Connection to Redis failed, will try to reconnect [attempt 2 of 10]
      

            vjuranek@redhat.com Vojtech Juranek
            vjuranek@redhat.com Vojtech Juranek
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: