Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-9943

Sometimes WF process does not exit on RHEL 6 (32/64 bit) and IBM JDK 8

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • None
    • 12.0.0.Final
    • Server
    • None
    • Hide

      Hard to reproduce – in less than 1% of cases in our testing environment

      Steps to reproduce: (RHEL 6, IBM SDK 8)
      Call :shutdown CLI operation on WF12 (started with default standalone-full.xml).
      Server logs that it stopped:

      2018-03-02 04:17:36,879 INFO  [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: JBoss EAP 7.2.0.EL12-Beta1 (WildFly Core 4.0.0.Beta1-redhat-1) stopped in 82ms

      but ps aux | grep java still shows that process is running:

      [hudson@rhel6-x86-medium-6746 ~]$ ps aux | grep java
hudson   25506  0.2  5.6 703128 227560 ?       Sl   04:17   0:09 /qa/tools/opt/ibm-java-80/bin/java -D[Standalone] -server -verbose:gc -Xloggc:/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms512m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.default.multicast.address=227.0.130.48 -Djboss.messaging.cluster.password=myGreatPasswd -Djboss.messaging.group.address=227.0.130.48 -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Dorg.jboss.boot.log.file=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/log/server.log -Dlogging.configuration=file:/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/configuration/logging.properties -jar /home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/jboss-modules.jar -mp /home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/modules org.jboss.as.standalone -Djboss.home.dir=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2 -Djboss.server.base.dir=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone -Djboss.bind.address=172.16.130.48 -Djboss.bind.address.unsecure=172.16.130.48 -Djboss.bind.address.management=172.16.130.48 -Djboss.bind.address.private=172.16.130.48 -c standalone-full.xml
      Show
      Hard to reproduce – in less than 1% of cases in our testing environment Steps to reproduce: (RHEL 6, IBM SDK 8) Call :shutdown CLI operation on WF12 (started with default standalone-full.xml). Server logs that it stopped: 2018-03-02 04:17:36,879 INFO [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: JBoss EAP 7.2.0.EL12-Beta1 (WildFly Core 4.0.0.Beta1-redhat-1) stopped in 82ms but ps aux | grep java still shows that process is running: [hudson@rhel6-x86-medium-6746 ~]$ ps aux | grep java hudson 25506 0.2 5.6 703128 227560 ? Sl 04:17 0:09 /qa/tools/opt/ibm-java-80/bin/java -D[Standalone] -server -verbose:gc -Xloggc:/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms512m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack= true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless= true -Djboss. default .multicast.address=227.0.130.48 -Djboss.messaging.cluster.password=myGreatPasswd -Djboss.messaging.group.address=227.0.130.48 -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=UTF-8 -Djava.net.preferIPv4Stack= true -Dorg.jboss.boot.log.file=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/log/server.log -Dlogging.configuration=file:/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone/configuration/logging.properties -jar /home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/jboss-modules.jar -mp /home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/modules org.jboss.as.standalone -Djboss.home.dir=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2 -Djboss.server.base.dir=/home/hudson/hudson_workspace/workspace/eap-7.x-acceptance-multinode-rhel-mnovak/4a508515/manu-eap-1.1.91/out/downloadEap1/workspace/jboss-eap-7.2/standalone -Djboss.bind.address=172.16.130.48 -Djboss.bind.address.unsecure=172.16.130.48 -Djboss.bind.address.management=172.16.130.48 -Djboss.bind.address. private =172.16.130.48 -c standalone-full.xml

      Sometimes happens that WF12 process does not exit when :shutdown CLI operation called and hangs indefinitely. This happens only RHEL 6 (32/64 bit) with IBM SDK8.

      Calling kill -3 does not create any javacore dump and only output could be gathered from pstack (attached). It indicates that JVM shutdown thread is hanging on:

      Thread 25 (Thread 0xb7792b70 (LWP 25507)):
#0  0x00bd4424 in __kernel_vsyscall ()
#1  0x00b2243c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x004f8b1f in monitor_wait_original () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9thr29.so
#3  0x004f9957 in omrthread_monitor_wait () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9thr29.so
#4  0x0033e217 in protectedDestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9vm29.so
#5  0x0042cf6e in omrsig_protect () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9prt29.so
#6  0x0033db93 in DestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libj9vm29.so
#7  0x00815999 in DestroyJavaVM () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/lib/i386/default/libjvm.so
#8  0x00528649 in JavaMain () from /qa/tools/opt/ibm-java-i386-sdk-8.0-5.10/jre/bin/../lib/i386/jli/libjli.so
#9  0x00b1ebc9 in start_thread () from /lib/libpthread.so.0
#10 0x001fc04e in clone () from /lib/libc.so.6

      There are other 24 threads which might be blocking this thread or a dead lock. This appears to be problem in IBM SDK 8 in combination RHEL 6 GLIBC_2.3.2 library.

        1. pstack.txt
          30 kB

              jgreene@redhat.com Jason Greene
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: