running low-tech devices:

JDK1.8.0_192,  wildfly10

server version:

Linux VM-0-6-ubuntu 4.15.0-159-generic #167-Ubuntu SMP Tue Sep 21 08:55:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Problem description:

The wildfly process still exists, but the website cannot be accessed, but the CPU usage exceeds 100%

Arthas can run, but cannot connect to the wildfly process

In the bin directory of wildfly, there is an hprof file

Before the problem occurred, it took more than ten days to run for a FULL GC to occur, but after the problem occurred, several FULL GCs could occur every minute.

During the period when the problem occurred, no large number of requests were found, only a small number of instruction requests were found, as well as socket connections (400 ports were opened, but only one thread was accessing)

And it's not that the CPU skyrocketed at the beginning and couldn't be accessed, it's only after a period of time that it couldn't be accessed.

Using pstack to view thread stack information：

#0  0x00007f5926773ad3 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f592019737c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f5920197328, cond=0x7f5920197350) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f5920197350, mutex=0x7f5920197328) at pthread_cond_wait.c:655
#3  0x00007f592588704b in os::PlatformEvent::park() () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#4  0x00007f592583c8b9 in Monitor::IWait(Thread*, long) () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#5  0x00007f592583de1a in Monitor::wait(bool, long, bool) () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#6  0x00007f5925a63b32 in FlexibleWorkGang::run_task(AbstractGangTask*) () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#7  0x00007f5925503863 in G1CollectedHeap::do_collection(bool, bool, unsigned long) [clone .part.286] () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#8  0x00007f5925504005 in G1CollectedHeap::satisfy_failed_allocation(unsigned long, unsigned char, bool*) () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#9  0x00007f5925a4afef in VM_G1CollectForAllocation::doit() () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#10 0x00007f5925a4a026 in VM_Operation::evaluate() () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#11 0x00007f5925a483ad in VMThread::evaluate_operation(VM_Operation*) [clone .constprop.44] () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#12 0x00007f5925a4896b in VMThread::loop() () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#13 0x00007f5925a48d18 in VMThread::run() () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#14 0x00007f592587e8b2 in java_start(Thread*) () from /usr/tools/jdk1.8.0_192/jre/lib/amd64/server/libjvm.so
#15 0x00007f592676d6db in start_thread (arg=0x7f58d0406700) at pthread_create.c:463
#16 0x00007f592607a61f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

execute command jstat -gcutil 23682 1000 10

  S0     S1     E      O      M     CCS    YGC     YGCT    FGC    FGCT     GCT   
  0.00   0.00   0.00  99.99  93.85  88.52   5762  426.398   385 1721.964 2148.363
  0.00   0.00   0.00  99.99  93.85  88.52   5762  426.398   385 1721.964 2148.363
  0.00   0.00   0.00  99.99  93.85  88.52   5762  426.398   385 1721.964 2148.363
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   386 1726.344 2152.862
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   386 1726.344 2152.862
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   386 1726.344 2152.862
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   386 1726.344 2152.862
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   387 1730.640 2157.158
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   387 1730.640 2157.158
  0.00   0.00   0.00  99.99  93.85  88.52   5764  426.518   387 1730.640 2157.158

this is gc log, gc.log

wildfly create heapdump.phrof, using MAT analysis:

The thread org.xnio.nio.WorkerThread @ 0xa0812858 default I/O-4 keeps local variables with total size 1,431,839,656 (85.77%) bytes.
The memory is accumulated in one instance of "java.lang.Object[]" loaded by "<system class loader>".

default I/O-4
  at sun.security.ssl.SSLEngineImpl.readNetRecord(Lsun/security/ssl/EngineArgs;)Ljavax/net/ssl/SSLEngineResult; (SSLEngineImpl.java:885)
  at sun.security.ssl.SSLEngineImpl.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (SSLEngineImpl.java:766)
  at javax.net.ssl.SSLEngine.unwrap(Ljava/nio/ByteBuffer;Ljava/nio/ByteBuffer;)Ljavax/net/ssl/SSLEngineResult; (SSLEngine.java:624)
  at io.undertow.protocols.ssl.SslConduit.doUnwrap([Ljava/nio/ByteBuffer;II)J (SslConduit.java:705)
  at io.undertow.protocols.ssl.SslConduit.doHandshake()V (SslConduit.java:608)
  at io.undertow.protocols.ssl.SslConduit.access$600(Lio/undertow/protocols/ssl/SslConduit;)V (SslConduit.java:63)
  at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady()V (SslConduit.java:1034)
  at io.undertow.protocols.ssl.SslConduit$1.run()V (SslConduit.java:229)
  at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V (WorkerThread.java:580)
  at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:464)

after using java VisualVM loading heapdump.phrof, Obtained the following information：

More than 90% of the total number of instances in the io. undertow. protocols. ssl. SslConduit $1 instance stack.