Blocker maschmid@redhat.com reported JWS-1397 with Tomcat on OpenShift, although the issue is not specific to OpenShift.
Follows a dead simple reproducer for local usage, independent of OpenShift/Docker:
Reproducer on JWS 5.1, Tomcat 9.0.7, tcnative 1.2.17, OpenJDK 1.8.0.212.b04-3.el8.x86_64
- Fetch and unzip Tomcat
- Fetch and unzip Jolokia
- Fetch ca.cert.crt
- Prepare jolokia.properties e.g. as in
host=127.0.0.1 port=8778 discoveryEnabled=false user=jolokia password=1dyar6DPuP65QYOu2hYXtZ3kunUcbr protocol=https useSslClientAuthentication=true extraClientCheck=true caCert=/opt/noe-tests/resources/ssl/proper/generated/ca/certs/ca.cert.crt
- In your Tomcat's bin/ directory:
- export CATALINA_OPTS="-Djava.security.egd=file:/dev/./urandom -javaagent:/home/karm/JWS/jolokia-1.6.1/agents/jolokia-jvm.jar=config=/home/karm/JWS/jolokia.properties"
- Use e.g. one-liner such as:
while [ 1 ];do sleep 1;pkill -TERM java;./catalina.sh start > /dev/null; until $(curl --silent --output /dev/null --fail --head http://127.0.0.1:8080/);do echo -e ".\c";sleep 0.5;done;echo OK; ./catalina.sh stop > /dev/null;done;
- Output could be as:
..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..OK ..............................................................
When the many dots appear, the Tomcat is stuck.
Thread dump
[root@rhel8 noe-tests]# jps 13719 Jps 13449 Bootstrap [root@rhel8 noe-tests]# jstack -F 13449 Attaching to process ID 13449, please wait... Debugger attached successfully. Server compiler detected. JVM version is 25.212-b04 Deadlock Detection: No deadlocks found. Thread 13469: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame) Thread 13468: (state = BLOCKED) - java.lang.Runtime.loadLibrary0(java.lang.Class, java.lang.String) @bci=0, line=862 (Interpreted frame) - java.lang.System.loadLibrary(java.lang.String) @bci=7, line=1122 (Interpreted frame) - sun.nio.fs.UnixNativeDispatcher$1.run() @bci=2, line=573 (Interpreted frame) - sun.nio.fs.UnixNativeDispatcher$1.run() @bci=1, line=571 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame) - sun.nio.fs.UnixNativeDispatcher.<clinit>() @bci=7, line=571 (Interpreted frame) - sun.nio.fs.UnixFileSystem.<init>(sun.nio.fs.UnixFileSystemProvider, java.lang.String) @bci=92, line=67 (Interpreted frame) - sun.nio.fs.LinuxFileSystem.<init>(sun.nio.fs.UnixFileSystemProvider, java.lang.String) @bci=3, line=39 (Interpreted frame) - sun.nio.fs.LinuxFileSystemProvider.newFileSystem(java.lang.String) @bci=6, line=46 (Interpreted frame) - sun.nio.fs.LinuxFileSystemProvider.newFileSystem(java.lang.String) @bci=2, line=39 (Interpreted frame) - sun.nio.fs.UnixFileSystemProvider.<init>() @bci=13, line=56 (Interpreted frame) - sun.nio.fs.LinuxFileSystemProvider.<init>() @bci=1, line=41 (Interpreted frame) - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=85, line=62 (Interpreted frame) - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame) - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=423 (Interpreted frame) - java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame) - sun.nio.fs.DefaultFileSystemProvider.createProvider(java.lang.String) @bci=19, line=48 (Interpreted frame) - sun.nio.fs.DefaultFileSystemProvider.create() @bci=42, line=63 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider() @bci=0, line=108 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder.access$000() @bci=0, line=89 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder$1.run() @bci=0, line=98 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder$1.run() @bci=1, line=96 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem() @bci=7, line=96 (Interpreted frame) - java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>() @bci=0, line=90 (Interpreted frame) - java.nio.file.FileSystems.getDefault() @bci=0, line=176 (Interpreted frame) - java.io.File.toPath() @bci=22, line=2234 (Interpreted frame) - org.apache.juli.FileHandler.streamFilesForDelete() @bci=26, line=541 (Interpreted frame) - org.apache.juli.FileHandler.lambda$clean$0() @bci=1, line=528 (Interpreted frame) - org.apache.juli.FileHandler$$Lambda$1.run() @bci=4 (Interpreted frame) - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Interpreted frame) - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1149 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame) Thread 13467: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Interpreted frame) - java.util.concurrent.LinkedBlockingDeque.pollFirst(long, java.util.concurrent.TimeUnit) @bci=52, line=522 (Interpreted frame) - java.util.concurrent.LinkedBlockingDeque.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=684 (Interpreted frame) - org.apache.juli.AsyncFileHandler$LoggerThread.run() @bci=10, line=160 (Interpreted frame) Thread 13461: (state = BLOCKED) - java.nio.file.FileSystems.getDefault() @bci=0, line=176 (Interpreted frame) - java.nio.file.Paths.get(java.net.URI) @bci=28, line=138 (Interpreted frame) - sun.misc.Launcher$ExtClassLoader.findLibrary(java.lang.String) @bci=41, line=235 (Interpreted frame) - java.lang.ClassLoader.loadLibrary(java.lang.Class, java.lang.String, boolean) @bci=88, line=1830 (Interpreted frame) - java.lang.Runtime.loadLibrary0(java.lang.Class, java.lang.String) @bci=54, line=870 (Interpreted frame) - java.lang.System.loadLibrary(java.lang.String) @bci=7, line=1122 (Interpreted frame) - sun.security.ec.SunEC$1.run() @bci=2, line=60 (Interpreted frame) - sun.security.ec.SunEC$1.run() @bci=1, line=58 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame) - sun.security.ec.SunEC.<clinit>() @bci=11, line=58 (Interpreted frame) - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=85, line=62 (Interpreted frame) - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame) - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=423 (Interpreted frame) - java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame) - sun.security.jca.ProviderConfig$2.run() @bci=79, line=221 (Interpreted frame) - sun.security.jca.ProviderConfig$2.run() @bci=1, line=206 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame) - sun.security.jca.ProviderConfig.doLoadProvider() @bci=8, line=206 (Interpreted frame) - sun.security.jca.ProviderConfig.getProvider() @bci=88, line=187 (Interpreted frame) - sun.security.jca.ProviderList.getProvider(int) @bci=6, line=233 (Interpreted frame) - sun.security.jca.ProviderList.getService(java.lang.String, java.lang.String) @bci=13, line=331 (Interpreted frame) - sun.security.jca.GetInstance.getInstance(java.lang.String, java.lang.Class, java.lang.String) @bci=7, line=157 (Interpreted frame) - javax.net.ssl.SSLContext.getInstance(java.lang.String) @bci=5, line=156 (Interpreted frame) - org.jolokia.jvmagent.JolokiaServer.createHttpsServer(java.net.InetSocketAddress, org.jolokia.jvmagent.JolokiaServerConfig) @bci=13, line=263 (Interpreted frame) - org.jolokia.jvmagent.JolokiaServer.createHttpServer(org.jolokia.jvmagent.JolokiaServerConfig) @bci=32, line=237 (Interpreted frame) - org.jolokia.jvmagent.JolokiaServer.init(org.jolokia.jvmagent.JolokiaServerConfig, boolean) @bci=3, line=167 (Interpreted frame) - org.jolokia.jvmagent.JolokiaServer.<init>(org.jolokia.jvmagent.JolokiaServerConfig, boolean) @bci=12, line=77 (Interpreted frame) - org.jolokia.jvmagent.JvmAgent$1.run() @bci=23, line=98 (Interpreted frame) Thread 13460: (state = BLOCKED) Thread 13459: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Interpreted frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=216 (Interpreted frame) Thread 13458: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame) - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Interpreted frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted frame) Thread 13451: (state = BLOCKED) - java.lang.Runtime.load0(java.lang.Class, java.lang.String) @bci=0, line=801 (Interpreted frame) - java.lang.System.load(java.lang.String) @bci=7, line=1086 (Interpreted frame) - org.apache.tomcat.jni.Library.<init>() @bci=65, line=42 (Interpreted frame) - org.apache.tomcat.jni.Library.initialize(java.lang.String) @bci=14, line=206 (Interpreted frame) - org.apache.catalina.core.AprLifecycleListener.init() @bci=30, line=198 (Interpreted frame) - org.apache.catalina.core.AprLifecycleListener.isAprAvailable() @bci=12, line=107 (Interpreted frame) - org.apache.catalina.connector.Connector.<init>(java.lang.String) @bci=119, line=79 (Interpreted frame) - org.apache.catalina.startup.ConnectorCreateRule.begin(java.lang.String, java.lang.String, org.xml.sax.Attributes) @bci=55, line=64 (Interpreted frame) - org.apache.tomcat.util.digester.Digester.startElement(java.lang.String, java.lang.String, java.lang.String, org.xml.sax.Attributes) @bci=325, line=1188 (Compiled frame) - com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(com.sun.org.apache.xerces.internal.xni.QName, com.sun.org.apache.xerces.internal.xni.XMLAttributes, com.sun.org.apache.xerces.internal.xni.Augmentations) @bci=297, line=509 (Compiled frame) - com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(com.sun.org.apache.xerces.internal.xni.QName, com.sun.org.apache.xerces.internal.xni.XMLAttributes, com.sun.org.apache.xerces.internal.xni.Augmentations) @bci=4, line=182 (Interpreted frame) - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement() @bci=378, line=1339 (Compiled frame) - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next() @bci=439, line=2784 (Compiled frame) - com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next() @bci=4, line=602 (Compiled frame) - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(boolean) @bci=308, line=505 (Interpreted frame) - com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(boolean) @bci=123, line=842 (Interpreted frame) - com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(com.sun.org.apache.xerces.internal.xni.parser.XMLInputSource) @bci=29, line=771 (Interpreted frame) - com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(com.sun.org.apache.xerces.internal.xni.parser.XMLInputSource) @bci=76, line=141 (Interpreted frame) - com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(org.xml.sax.InputSource) @bci=43, line=1213 (Interpreted frame) - com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(org.xml.sax.InputSource) @bci=53, line=643 (Interpreted frame) - org.apache.tomcat.util.digester.Digester.parse(org.xml.sax.InputSource) @bci=9, line=1453 (Interpreted frame) - org.apache.catalina.startup.Catalina.load() @bci=445, line=605 (Interpreted frame) - org.apache.catalina.startup.Catalina.load(java.lang.String[]) @bci=9, line=656 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame) - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame) - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame) - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame) - org.apache.catalina.startup.Bootstrap.load(java.lang.String[]) @bci=104, line=306 (Interpreted frame) - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=164, line=491 (Interpreted frame)
Possibly related to
Fixed upstream
I couldn't reproduce it with Tomcat 9.0.20 / tcnative 1.2.17. That matches with Mark's note:
Fixed in: - master for 9.0.18 onwards