Uploaded image for project: 'JBoss Web Server'
  1. JBoss Web Server
  2. JWS-1414

Tomcat frequently hangs at startup when Jolokia loads certificate [jws-3]

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 3.1.7.DR2
    • 3.1.7.DR1, 5.1.0.GA
    • tomcat7, tomcat8
    • 4
    • +

    Description

      maschmid@redhat.com reported JWS-1397 with Tomcat on OpenShift, although the issue is not specific to OpenShift.
      Follows a dead simple reproducer for local usage, independent of OpenShift/Docker:

      Reproducer on JWS 5.1, Tomcat 9.0.7, tcnative 1.2.17, OpenJDK 1.8.0.212.b04-3.el8.x86_64

      1. Fetch and unzip Tomcat
      2. Fetch and unzip Jolokia
      3. Fetch ca.cert.crt
      4. Prepare jolokia.properties e.g. as in 
        host=127.0.0.1
port=8778
discoveryEnabled=false
user=jolokia
password=1dyar6DPuP65QYOu2hYXtZ3kunUcbr
protocol=https
useSslClientAuthentication=true
extraClientCheck=true
caCert=/opt/noe-tests/resources/ssl/proper/generated/ca/certs/ca.cert.crt
      5. In your Tomcat's bin/ directory:
      6. export CATALINA_OPTS="-Djava.security.egd=file:/dev/./urandom -javaagent:/home/karm/JWS/jolokia-1.6.1/agents/jolokia-jvm.jar=config=/home/karm/JWS/jolokia.properties"
      7. Use e.g. one-liner such as: 
        while [ 1 ];do sleep 1;pkill -TERM java;./catalina.sh start > /dev/null;
   until $(curl --silent --output /dev/null --fail --head http://127.0.0.1:8080/);do echo -e ".\c";sleep 0.5;done;echo OK;
   ./catalina.sh stop  > /dev/null;done;
      8. Output could be as: 
        ..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..OK
..............................................................

        When the many dots appear, the Tomcat is stuck.

      Thread dump

      [root@rhel8 noe-tests]# jps
13719 Jps
13449 Bootstrap

[root@rhel8 noe-tests]# jstack -F 13449 
Attaching to process ID 13449, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.212-b04
Deadlock Detection:

No deadlocks found.

Thread 13469: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.util.TimerThread.mainLoop() @bci=201, line=552 (Interpreted frame)
 - java.util.TimerThread.run() @bci=1, line=505 (Interpreted frame)


Thread 13468: (state = BLOCKED)
 - java.lang.Runtime.loadLibrary0(java.lang.Class, java.lang.String) @bci=0, line=862 (Interpreted frame)
 - java.lang.System.loadLibrary(java.lang.String) @bci=7, line=1122 (Interpreted frame)
 - sun.nio.fs.UnixNativeDispatcher$1.run() @bci=2, line=573 (Interpreted frame)
 - sun.nio.fs.UnixNativeDispatcher$1.run() @bci=1, line=571 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame)
 - sun.nio.fs.UnixNativeDispatcher.<clinit>() @bci=7, line=571 (Interpreted frame)
 - sun.nio.fs.UnixFileSystem.<init>(sun.nio.fs.UnixFileSystemProvider, java.lang.String) @bci=92, line=67 (Interpreted frame)
 - sun.nio.fs.LinuxFileSystem.<init>(sun.nio.fs.UnixFileSystemProvider, java.lang.String) @bci=3, line=39 (Interpreted frame)
 - sun.nio.fs.LinuxFileSystemProvider.newFileSystem(java.lang.String) @bci=6, line=46 (Interpreted frame)
 - sun.nio.fs.LinuxFileSystemProvider.newFileSystem(java.lang.String) @bci=2, line=39 (Interpreted frame)
 - sun.nio.fs.UnixFileSystemProvider.<init>() @bci=13, line=56 (Interpreted frame)
 - sun.nio.fs.LinuxFileSystemProvider.<init>() @bci=1, line=41 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=85, line=62 (Interpreted frame)
 - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame)
 - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=423 (Interpreted frame)
 - java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame)
 - sun.nio.fs.DefaultFileSystemProvider.createProvider(java.lang.String) @bci=19, line=48 (Interpreted frame)
 - sun.nio.fs.DefaultFileSystemProvider.create() @bci=42, line=63 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder.getDefaultProvider() @bci=0, line=108 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder.access$000() @bci=0, line=89 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder$1.run() @bci=0, line=98 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder$1.run() @bci=1, line=96 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder.defaultFileSystem() @bci=7, line=96 (Interpreted frame)
 - java.nio.file.FileSystems$DefaultFileSystemHolder.<clinit>() @bci=0, line=90 (Interpreted frame)
 - java.nio.file.FileSystems.getDefault() @bci=0, line=176 (Interpreted frame)
 - java.io.File.toPath() @bci=22, line=2234 (Interpreted frame)
 - org.apache.juli.FileHandler.streamFilesForDelete() @bci=26, line=541 (Interpreted frame)
 - org.apache.juli.FileHandler.lambda$clean$0() @bci=1, line=528 (Interpreted frame)
 - org.apache.juli.FileHandler$$Lambda$1.run() @bci=4 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511 (Interpreted frame)
 - java.util.concurrent.FutureTask.run() @bci=42, line=266 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1149 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=624 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)


Thread 13467: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingDeque.pollFirst(long, java.util.concurrent.TimeUnit) @bci=52, line=522 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingDeque.poll(long, java.util.concurrent.TimeUnit) @bci=3, line=684 (Interpreted frame)
 - org.apache.juli.AsyncFileHandler$LoggerThread.run() @bci=10, line=160 (Interpreted frame)


Thread 13461: (state = BLOCKED)
 - java.nio.file.FileSystems.getDefault() @bci=0, line=176 (Interpreted frame)
 - java.nio.file.Paths.get(java.net.URI) @bci=28, line=138 (Interpreted frame)
 - sun.misc.Launcher$ExtClassLoader.findLibrary(java.lang.String) @bci=41, line=235 (Interpreted frame)
 - java.lang.ClassLoader.loadLibrary(java.lang.Class, java.lang.String, boolean) @bci=88, line=1830 (Interpreted frame)
 - java.lang.Runtime.loadLibrary0(java.lang.Class, java.lang.String) @bci=54, line=870 (Interpreted frame)
 - java.lang.System.loadLibrary(java.lang.String) @bci=7, line=1122 (Interpreted frame)
 - sun.security.ec.SunEC$1.run() @bci=2, line=60 (Interpreted frame)
 - sun.security.ec.SunEC$1.run() @bci=1, line=58 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame)
 - sun.security.ec.SunEC.<clinit>() @bci=11, line=58 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=85, line=62 (Interpreted frame)
 - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame)
 - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=423 (Interpreted frame)
 - java.lang.Class.newInstance() @bci=138, line=442 (Interpreted frame)
 - sun.security.jca.ProviderConfig$2.run() @bci=79, line=221 (Interpreted frame)
 - sun.security.jca.ProviderConfig$2.run() @bci=1, line=206 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedAction) @bci=0 (Interpreted frame)
 - sun.security.jca.ProviderConfig.doLoadProvider() @bci=8, line=206 (Interpreted frame)
 - sun.security.jca.ProviderConfig.getProvider() @bci=88, line=187 (Interpreted frame)
 - sun.security.jca.ProviderList.getProvider(int) @bci=6, line=233 (Interpreted frame)
 - sun.security.jca.ProviderList.getService(java.lang.String, java.lang.String) @bci=13, line=331 (Interpreted frame)
 - sun.security.jca.GetInstance.getInstance(java.lang.String, java.lang.Class, java.lang.String) @bci=7, line=157 (Interpreted frame)
 - javax.net.ssl.SSLContext.getInstance(java.lang.String) @bci=5, line=156 (Interpreted frame)
 - org.jolokia.jvmagent.JolokiaServer.createHttpsServer(java.net.InetSocketAddress, org.jolokia.jvmagent.JolokiaServerConfig) @bci=13, line=263 (Interpreted frame)
 - org.jolokia.jvmagent.JolokiaServer.createHttpServer(org.jolokia.jvmagent.JolokiaServerConfig) @bci=32, line=237 (Interpreted frame)
 - org.jolokia.jvmagent.JolokiaServer.init(org.jolokia.jvmagent.JolokiaServerConfig, boolean) @bci=3, line=167 (Interpreted frame)
 - org.jolokia.jvmagent.JolokiaServer.<init>(org.jolokia.jvmagent.JolokiaServerConfig, boolean) @bci=12, line=77 (Interpreted frame)
 - org.jolokia.jvmagent.JvmAgent$1.run() @bci=23, line=98 (Interpreted frame)


Thread 13460: (state = BLOCKED)


Thread 13459: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=216 (Interpreted frame)


Thread 13458: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame)
 - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted frame)


Thread 13451: (state = BLOCKED)
 - java.lang.Runtime.load0(java.lang.Class, java.lang.String) @bci=0, line=801 (Interpreted frame)
 - java.lang.System.load(java.lang.String) @bci=7, line=1086 (Interpreted frame)
 - org.apache.tomcat.jni.Library.<init>() @bci=65, line=42 (Interpreted frame)
 - org.apache.tomcat.jni.Library.initialize(java.lang.String) @bci=14, line=206 (Interpreted frame)
 - org.apache.catalina.core.AprLifecycleListener.init() @bci=30, line=198 (Interpreted frame)
 - org.apache.catalina.core.AprLifecycleListener.isAprAvailable() @bci=12, line=107 (Interpreted frame)
 - org.apache.catalina.connector.Connector.<init>(java.lang.String) @bci=119, line=79 (Interpreted frame)
 - org.apache.catalina.startup.ConnectorCreateRule.begin(java.lang.String, java.lang.String, org.xml.sax.Attributes) @bci=55, line=64 (Interpreted frame)
 - org.apache.tomcat.util.digester.Digester.startElement(java.lang.String, java.lang.String, java.lang.String, org.xml.sax.Attributes) @bci=325, line=1188 (Compiled frame)
 - com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(com.sun.org.apache.xerces.internal.xni.QName, com.sun.org.apache.xerces.internal.xni.XMLAttributes, com.sun.org.apache.xerces.internal.xni.Augmentations) @bci=297, line=509 (Compiled frame)
 - com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(com.sun.org.apache.xerces.internal.xni.QName, com.sun.org.apache.xerces.internal.xni.XMLAttributes, com.sun.org.apache.xerces.internal.xni.Augmentations) @bci=4, line=182 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement() @bci=378, line=1339 (Compiled frame)
 - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next() @bci=439, line=2784 (Compiled frame)
 - com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next() @bci=4, line=602 (Compiled frame)
 - com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(boolean) @bci=308, line=505 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(boolean) @bci=123, line=842 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(com.sun.org.apache.xerces.internal.xni.parser.XMLInputSource) @bci=29, line=771 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(com.sun.org.apache.xerces.internal.xni.parser.XMLInputSource) @bci=76, line=141 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(org.xml.sax.InputSource) @bci=43, line=1213 (Interpreted frame)
 - com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(org.xml.sax.InputSource) @bci=53, line=643 (Interpreted frame)
 - org.apache.tomcat.util.digester.Digester.parse(org.xml.sax.InputSource) @bci=9, line=1453 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load() @bci=445, line=605 (Interpreted frame)
 - org.apache.catalina.startup.Catalina.load(java.lang.String[]) @bci=9, line=656 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke0(java.lang.reflect.Method, java.lang.Object, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=100, line=62 (Interpreted frame)
 - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Interpreted frame)
 - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=56, line=498 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.load(java.lang.String[]) @bci=104, line=306 (Interpreted frame)
 - org.apache.catalina.startup.Bootstrap.main(java.lang.String[]) @bci=164, line=491 (Interpreted frame)

      Possibly related to

      Fixed upstream

      I couldn't reproduce it with Tomcat 9.0.20 / tcnative 1.2.17. That matches with Mark's note:

      Fixed in:
  - master for 9.0.18 onwards

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JWS-1401 Tomcat frequently hangs at startup when Jolokia loads certificate [jws-5]

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              rhn-support-csutherl Coty Sutherland
              mbabacek1@redhat.com Michal Karm
              Jan Onderka Jan Onderka
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: