Loading...

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: None
Affects Version/s: 1.4.1.GA, 1.6.2.GA
Component/s: kafka-connect
Labels:
None

Blocked:
False
Ready:
False
GSS Priority:
Release Note Text:
Undefined
Target Release:

1.6.4.GA
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Class loader level deadlock when having more than one Connect worker node and starting a bunch of connectors with their SMTs. It seems to be much more likely on an idle cluster (with all connectors in a paused state). We confirmed it's the same issue of the upstream KAFKA-7421 by taking thread dumps from worker pods.

In this case, it happens frequently when deleting a worker pod or a rolling update starts because of a configuration change. In both cases all Connect workers blocks because of the deadlock and the rebalance process hangs. From logs you see that all workers are at different generations and unable to join the cluster. Deadlocks are also detected by the jvm_threads_deadlocked_monitor metric.

Found one Java-level deadlock:
=============================
"StartAndStopExecutor-connect-1-8":
  waiting to lock monitor 0x00007f3bc4004458 (object 0x0000000085220f10, a org.apache.kafka.connect.runtime.isolation.PluginClassLoader),
  which is held by "StartAndStopExecutor-connect-1-7"
"StartAndStopExecutor-connect-1-7":
  waiting to lock monitor 0x00007f3be8009c78 (object 0x0000000085214aa0, a org.apache.kafka.connect.runtime.isolation.PluginClassLoader),
  which is held by "StartAndStopExecutor-connect-1-1"
"StartAndStopExecutor-connect-1-1":
  waiting to lock monitor 0x00007f3bc4006fa8 (object 0x000000008007d7d8, a org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader),
  which is held by "StartAndStopExecutor-connect-1-4"
"StartAndStopExecutor-connect-1-4":
  waiting to lock monitor 0x00007f3be8009c78 (object 0x0000000085214aa0, a org.apache.kafka.connect.runtime.isolation.PluginClassLoader),
  which is held by "StartAndStopExecutor-connect-1-1"

Java stack information for the threads listed above:
===================================================
"StartAndStopExecutor-connect-1-8":
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:91)
	- waiting to lock <0x0000000085220f10> (a org.apache.kafka.connect.runtime.isolation.PluginClassLoader)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:716)
	at org.apache.kafka.connect.runtime.ConnectorConfig.enrich(ConnectorConfig.java:308)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:212)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:206)
	at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:430)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1147)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1600(DistributedHerder.java:126)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1162)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1158)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"StartAndStopExecutor-connect-1-7":
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:91)
	- waiting to lock <0x0000000085214aa0> (a org.apache.kafka.connect.runtime.isolation.PluginClassLoader)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:394)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
	- locked <0x00000000f9f80b98> (a java.lang.Object)
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
	- locked <0x00000000f9f80b98> (a java.lang.Object)
	- locked <0x0000000085220f10> (a org.apache.kafka.connect.runtime.isolation.PluginClassLoader)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:716)
	at org.apache.kafka.connect.runtime.ConnectorConfig.enrich(ConnectorConfig.java:308)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:212)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:206)
	at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:430)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1147)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1600(DistributedHerder.java:126)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1162)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$12.call(DistributedHerder.java:1158)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"StartAndStopExecutor-connect-1-1":
	at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
	- waiting to lock <0x000000008007d7d8> (a org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:397)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
	- locked <0x00000000f9f823c0> (a java.lang.Object)
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:104)
	- locked <0x00000000f9f823c0> (a java.lang.Object)
	- locked <0x0000000085214aa0> (a org.apache.kafka.connect.runtime.isolation.PluginClassLoader)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:719)
	at org.apache.kafka.connect.runtime.ConnectorConfig.enrich(ConnectorConfig.java:308)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:212)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:206)
	at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:249)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:1190)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1300(DistributedHerder.java:126)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$14.call(DistributedHerder.java:1206)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$14.call(DistributedHerder.java:1202)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
"StartAndStopExecutor-connect-1-4":
	at org.apache.kafka.connect.runtime.isolation.PluginClassLoader.loadClass(PluginClassLoader.java:91)
	- waiting to lock <0x0000000085214aa0> (a org.apache.kafka.connect.runtime.isolation.PluginClassLoader)
	at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:394)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:719)
	at org.apache.kafka.connect.runtime.ConnectorConfig.enrich(ConnectorConfig.java:308)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:212)
	at org.apache.kafka.connect.runtime.ConnectorConfig.<init>(ConnectorConfig.java:206)
	at org.apache.kafka.connect.runtime.Worker.startConnector(Worker.java:249)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startConnector(DistributedHerder.java:1190)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1300(DistributedHerder.java:126)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$14.call(DistributedHerder.java:1206)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$14.call(DistributedHerder.java:1202)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

clones

ENTMQST-2639 Deadlock in Kafka Connect

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates