-
Bug
-
Resolution: Won't Do
-
Major
-
None
-
None
-
None
-
None
Running Strimzi on an OCP4 cluster (running on AWS) during the RH Summit 2019 demo developing it seems that after doing a rolling update of the Kafka brokers (due to an updated configuration), the TO isn't able to reconnect anymore.
[2019-04-12 08:34:30,851] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 2 (demo2019-kafka-2.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.129.2.53:9091) could not be established. Broker may not be available.
[2019-04-12 08:34:33,923] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 0 (demo2019-kafka-0.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.128.4.60:9091) could not be established. Broker may not be available.
[2019-04-12 08:34:36,995] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 2 (demo2019-kafka-2.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.129.2.53:9091) could not be established. Broker may not be available.
[2019-04-12 08:34:40,067] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 0 (demo2019-kafka-0.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.128.4.60:9091) could not be established. Broker may not be available.
[2019-04-12 08:34:43,139] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 2 (demo2019-kafka-2.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.129.2.53:9091) could not be established. Broker may not be available.
[2019-04-12 08:34:46,211] WARN <etworkClient:721> [dminclient-1] [AdminClient clientId=adminclient-1] Connection to node 0 (demo2019-kafka-0.demo2019-kafka-brokers.strimzi-demo-test.svc.cluster.local/10.128.4.60:9091) could not be established. Broker may not be available.
The TO is still using the old IPs assigned to the Kafka brokers at the deployment time and not the new ones after rolling update.
It's not an OCP DNS problem because new clients can connect to the brokers and even scaling the TO down/up works fine.
It should a DNS caching problem at AdminClient level or JVM level.