XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: None
Affects Version/s: 1.3.6.Final
Component/s: Core + SPIs
Labels:
None

There are two configurations debated on this JIRA, the first seems to work O.K. with some concerns, the second one doesn't work at all .

1. "ALPN hack", Oracle JDK8

Hello, when you have this configuration of Apache HTTP Server 2.4.23 mod_cluster.conf and EAP 7.1 standalone-ha.xml and you start both servers, the worker correctly contacts the balancer and it is able to serve requests via HTTP/2.0. Although, I am mildly concerned about the irregular messages I can see during httpd <-> eap periodic 5s liveness verification (LBstatusRecalTime) with OPTIONS method and about connector's exceptions during sending MCMP messages. I haven't tried to let it run for days, but long-time impact is my main motivation for posting this. I parsed the log into corresponding blocks to illustrate situation on the balancer side and on the worker side.

cap (wireshark)

This is the recording of the initial communication worker -> balancer (registering message), balancer -> worker (response): JBEAP-11548_good.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

Start

Both servers started, no client requests whatsoever. The first block is the very first communication between servers.

Worker Block 1

05:33:02,544 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
05:33:02,695 DEBUG [io.undertow.request] (default I/O-2) Using ALPN provider JDK8AlpnProvider for connector at /192.168.122.172:8443

Balancer Block 1

Worker Block 2

No exception, no log.

Balancer Block 2

Worker Block 3

05:33:23,144 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

Balancer Block 3

Worker Block 4

No exception, no log.

Balancer Block 4

Worker Block 5

No exception, no log.

Balancer Block 5

Worker Block 6

No exception, no log.

Balancer Block 6

Worker Block 7

No exception, no log.

Balancer Block 7

Worker Block 8

No exception, no log.

Balancer Block 8

Worker Block 9

No exception, no log.

Balancer Block 9

Worker Block 10

05:33:58,238 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

Balancer Block 10

Worker Block 11

05:34:02,808 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
05:34:03,002 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

Balancer Block 11

Worker Block 12

No exception, no log.

Balancer Block 12

Worker Block 13

05:34:13,242 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

Balancer Block 13

Worker Block 14

05:34:18,230 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

Balancer Block 14

Clients

Worker is correctly registered:

mod_cluster/1.3.5.Final
Node jboss-eap-7.1 (https://192.168.122.172:8443):
Balancer: qacluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 2,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 99
Virtual Host 1:
Contexts:
  /clusterbench, Status: ENABLED Request: 0 Disable Stop
  /, Status: ENABLED Request: 0 Disable Stop
Aliases:
  default-host
  localhost

Curl

Not related to EAP, nss Fedora 25 shenanigans with using blacklisted cipher suite: tls cipher ECDHE-RSA-AES256-SHA blacklisted by rfc7540

curl --http2 https://rhel7GAx86-64:2080/clusterbench/session --insecure -i -vvv, curl log
Worker - no log, wasn't actually called by the balancer, the request crashed earlier
Balancer - tells curl to go away, handshake log

Chrome

Chrome 58.0.3029.110 (64-bit), correctly display worker's web app on https://rhel7GAx86-64:2080/clusterbench/session via HTTP/2.0.
Worker - web app served, log
Balancer - O.K., log

Firefox

Firefox 53.0.3 (64-bit) correctly display worker's web app on https://rhel7GAx86-64:2080/clusterbench/session via HTTP/2.0.
Worker - O.K., log
Balancer - O.K., log

2. ALPN via Wildfly OpenSSL + OpenSSL 1.0.2h + Oracle JDK8

With the same Apache HTTP Server config as in the aforementioned case, i.e. mod_cluster.conf, and EAP 7.1 configured for Wildfly OpenSSL: standalone-ha.xml_openssl, the whole setup falls apart at the very beginning when worker contacts the balancer and then is unable to parse balancer's response. Noteworthy: If you put another EAP configured as an Undertow mod_cluster balancer in front of this worker, also with Wildfly OpenSSL, it works.

cap (wireshark)

This is the recording of the initial communication worker -> balancer (registering message), followed immediately by worker's crash during handshake, i.e. there is no balancer -> worker message: JBEAP-11548_bad.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

Balancer

Everything appears cool, mod_manager.c(3104): manager_handler INFO OK, see the log snippet.

Worker

Falls flat on its face while processing balancer's response:

06:16:24,031 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NumberFormatException: null
    at java.lang.Integer.parseInt(Integer.java:542)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:702)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
    at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454)
    at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)

Result

No worker is registered, integration is broken. The same worker setup with another Wildfly (EAP) Undertow mod_cluster balancer works.

WDYT guys, rhn-engineering-rhusar, sdouglas1@redhat.com, rhn-engineering-jclere?

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

JBEAP-11548_bad.pcap
11 kB
2017/06/15 5:33 AM
JBEAP-11548_good.pcap
58 kB
2017/06/15 5:33 AM

is cloned by

JBEAP-11548 HTTP/2: WildFly worker's interoperability with Apache HTTP Server mod_cluster

Closed

is related to

MODCLUSTER-800 Enable mod_proxy_cluster logic to correctly use the Apache httpd websocket feature introduced in 2.4.47

Resolved

Assignee:: Radoslav Husar

Reporter:: Karm Karm

Votes:: 1 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: 2017/06/14 8:47 AM

Updated:: 2024/07/02 10:17 AM