When I was testing executing WildFly workloads on a virtual thread, turning on JVM thread pinning logging (see WFLY-19536) revealed pinning happening in client-side Eclipse Angus code (e.g. sending emails). This issue is to trace working with the Angus community to resolve these issues and get the fixed code into WildFly.
See occurrences of 'onPinned' in https://ci.wildfly.org/repository/download/WF_MainNightlyJobs_StandardLinuxJdk21/446569:id/testsuite/integration/basic/target/wildfly/standalone/log/server.log. Lines in the stack trace annotated with "<== monitor" show where a monitor lock was taken that results in pinning when blocking code executes.
An example:
2024-07-18 05:23:38,520 INFO [stdout] (ForkJoinPool-1-worker-16) Thread[#18608,ForkJoinPool-1-worker-16,5,CarrierThreads] 2024-07-18 05:23:38,520 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.lang.VirtualThread$VThreadContinuation.onPinned(VirtualThread.java:185) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/jdk.internal.vm.Continuation.onPinned0(Continuation.java:393) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.lang.VirtualThread.park(VirtualThread.java:592) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.lang.System$2.parkVirtualThread(System.java:2639) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/jdk.internal.misc.VirtualThreads.park(VirtualThreads.java:54) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:369) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.Poller$Request.awaitFinish(Poller.java:215) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.Poller.pollIndirect(Poller.java:143) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.Poller.poll(Poller.java:102) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.Poller.poll(Poller.java:89) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:175) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:201) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:597) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.net.Socket.connect(Socket.java:751) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) java.base/java.net.Socket.connect(Socket.java:686) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) org.eclipse.angus.mail@2.0.3//org.eclipse.angus.mail.util.SocketFetcher.createSocket(SocketFetcher.java:368) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) org.eclipse.angus.mail@2.0.3//org.eclipse.angus.mail.util.SocketFetcher.getSocket(SocketFetcher.java:243) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) org.eclipse.angus.mail@2.0.3//org.eclipse.angus.mail.pop3.Protocol.<init>(Protocol.java:121) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) org.eclipse.angus.mail@2.0.3//org.eclipse.angus.mail.pop3.POP3Store.getPort(POP3Store.java:250) <== monitors:1 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) org.eclipse.angus.mail@2.0.3//org.eclipse.angus.mail.pop3.POP3Store.protocolConnect(POP3Store.java:192) <== monitors:1 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) jakarta.mail.api@2.1.3//jakarta.mail.Service.connect(Service.java:345) <== monitors:1 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) jakarta.mail.api@2.1.3//jakarta.mail.Service.connect(Service.java:225) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) jakarta.mail.api@2.1.3//jakarta.mail.Service.connect(Service.java:246) 2024-07-18 05:23:38,521 INFO [stdout] (ForkJoinPool-1-worker-16) deployment.d1e69a0b-f63d-4d5f-aa9a-5f42cf984c53.war//org.jboss.as.test.integration.mail.basic.MailTesterServlet.tryToReadEmails(MailTesterServlet.java:73)
Note that in the above stack, jakarta.mail.Service.connect also pins the thread. I'll file a separate issue for that.
To reproduce build https://github.com/bstansberry/wildfly/tree/pinning_check on SE 21.
- blocks
-
WFLY-19563 Support use of virtual threads in the Jakarta Concurrency 3.1. implementation
-
- Open
-
- is incorporated by
-
-
- Open
-
- is related to
-
WFLY-19713 Resolve virtual thread pinning issues in client side Jakarta Activation code
-
- Open
-
-
-
- Open
-
[WFLY-19711] Resolve virtual thread pinning issues in client side Angus code
bstansbe@redhat.com On Angus Mail, there is an initiative to support Virtual Threads with Jakarta Mail; this is being tracked here https://github.com/eclipse-ee4j/angus-mail/issues/128, but it seems they are just waiting for a solution at JDK level.
For Jakarta Mail, it seems a complex task since Jakarta Mail has a heavy usage of synchronized blocks and in any case, if that gets resolved it will be in a major version with backward incompatibilities changes, and it looks like this won't happen in Jakarta EE 11. As far as I can read, it looks like they are wanting to know what are the future plans for a JDK to support the synchronized blocks without pinning issues, it seems project Loom was working on it: https://mail.openjdk.org/pipermail/loom-dev/2024-February/006433.html
Brian Stansberry
added a comment - I've attached a file of lines grepped from https://ci.wildfly.org/repository/download/WF_MainNightlyJobs_StandardLinuxJdk21/446569:id/testsuite/integration/basic/target/wildfly/standalone/log/server.log that relate to Angus. At a glance it looks like there are 15-20 spots where pinning occurs spread over 5 or so classes.
Brian Stansberry
added a comment - I've attached a file of lines grepped from https://ci.wildfly.org/repository/download/WF_MainNightlyJobs_StandardLinuxJdk21/446569:id/testsuite/integration/basic/target/wildfly/standalone/log/server.log that relate to Angus. At a glance it looks like there are 15-20 spots where pinning occurs spread over 5 or so classes. Let's talk before doing too much here. The mail use case will be affected by similar problems in two Jakarta API artifacts. The urgency of fixing any particular part depends on the ability to solve the entire use case, which seems limited if public Jakarta APIs are involved.
Brian Stansberry
added a comment - - edited Let's talk before doing too much here. The mail use case will be affected by similar problems in two Jakarta API artifacts. The urgency of fixing any particular part depends on the ability to solve the entire use case, which seems limited if public Jakarta APIs are involved. 
Thanks for the update yborgess1@redhat.com. I think 'wait for Loom' will be a common response to this to topic, and when it comes to Jakarta APIs it's likely a necessary one, unless we were willing/able to fork the API projects. Which doesn't seem likely. And there is no point forking the API if the impl also has issues.