Uploaded image for project: 'mod_cluster'
  1. mod_cluster
  2. MODCLUSTER-588

HTTP/2.0: WildFly worker's interoperability with Apache HTTP Server mod_cluster

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Critical
    • 2.1.0.Final
    • 1.3.6.Final
    • Core + SPIs
    • None

    Description

      There are two configurations debated on this JIRA, the first seems to work O.K. with some concerns, the second one doesn't work at all .

      1. "ALPN hack", Oracle JDK8

      Hello, when you have this configuration of Apache HTTP Server 2.4.23 mod_cluster.conf and EAP 7.1 standalone-ha.xml and you start both servers, the worker correctly contacts the balancer and it is able to serve requests via HTTP/2.0. Although, I am mildly concerned about the irregular messages I can see during httpd <-> eap periodic 5s liveness verification (LBstatusRecalTime) with OPTIONS method and about connector's exceptions during sending MCMP messages. I haven't tried to let it run for days, but long-time impact is my main motivation for posting this. I parsed the log into corresponding blocks to illustrate situation on the balancer side and on the worker side.

      cap (wireshark)

      This is the recording of the initial communication worker -> balancer (registering message), balancer -> worker (response): JBEAP-11548_good.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

      Start

      Both servers started, no client requests whatsoever. The first block is the very first communication between servers.

      Worker Block 1

      05:33:02,544 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
05:33:02,695 DEBUG [io.undertow.request] (default I/O-2) Using ALPN provider JDK8AlpnProvider for connector at /192.168.122.172:8443

      Balancer Block 1

      Full block

      Worker Block 2

      No exception, no log.

      Balancer Block 2

      Full block

      Worker Block 3

      05:33:23,144 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 3

      Full block

      Worker Block 4

      No exception, no log.

      Balancer Block 4

      Full block

      Worker Block 5

      No exception, no log.

      Balancer Block 5

      Full block

      Worker Block 6

      No exception, no log.

      Balancer Block 6

      Full block

      Worker Block 7

      No exception, no log.

      Balancer Block 7

      Full block

      Worker Block 8

      No exception, no log.

      Balancer Block 8

      Full block

      Worker Block 9

      No exception, no log.

      Balancer Block 9

      Full block

      Worker Block 10

      05:33:58,238 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

      Full block

      Balancer Block 10

      Full block

      Worker Block 11

      05:34:02,808 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
05:34:03,002 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 11

      Full block

      Worker Block 12

      No exception, no log.

      Balancer Block 12

      Full block

      Worker Block 13

      05:34:13,242 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
    at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
    at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
    at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
    at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 13

      Full block

      Worker Block 14

      05:34:18,230 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

      Full block

      Balancer Block 14

      Full block

      Clients

      Worker is correctly registered:

      mod_cluster/1.3.5.Final
Node jboss-eap-7.1 (https://192.168.122.172:8443):
Balancer: qacluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 2,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 99
Virtual Host 1:
Contexts:
  /clusterbench, Status: ENABLED Request: 0 Disable Stop
  /, Status: ENABLED Request: 0 Disable Stop
Aliases:
  default-host
  localhost

      Curl

      Not related to EAP, nss Fedora 25 shenanigans with using blacklisted cipher suite: tls cipher ECDHE-RSA-AES256-SHA blacklisted by rfc7540

      Chrome

      Firefox

      2. ALPN via Wildfly OpenSSL + OpenSSL 1.0.2h + Oracle JDK8

      With the same Apache HTTP Server config as in the aforementioned case, i.e. mod_cluster.conf, and EAP 7.1 configured for Wildfly OpenSSL: standalone-ha.xml_openssl, the whole setup falls apart at the very beginning when worker contacts the balancer and then is unable to parse balancer's response. Noteworthy: If you put another EAP configured as an Undertow mod_cluster balancer in front of this worker, also with Wildfly OpenSSL, it works.

      cap (wireshark)

      This is the recording of the initial communication worker -> balancer (registering message), followed immediately by worker's crash during handshake, i.e. there is no balancer -> worker message: JBEAP-11548_bad.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

      Balancer

      Everything appears cool, mod_manager.c(3104): manager_handler INFO OK, see the log snippet.

      Worker

      Falls flat on its face while processing balancer's response: 

      06:16:24,031 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NumberFormatException: null
    at java.lang.Integer.parseInt(Integer.java:542)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:702)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387)
    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
    at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454)
    at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      , see full log

      Result

      No worker is registered, integration is broken. The same worker setup with another Wildfly (EAP) Undertow mod_cluster balancer works.

      WDYT guys, rhn-engineering-rhusar, sdouglas1@redhat.com, rhn-engineering-jclere?

      Attachments

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-11548 HTTP/2: WildFly worker's interoperability with Apache HTTP Server mod_cluster

          • Critical - To be worked on immediately following BLOCKER issues.
          • Verified
          is related to

          Feature Request - Feature requests from customers and/or users MODCLUSTER-800 Enable mod_proxy_cluster logic to correctly use the Apache httpd websocket feature introduced in 2.4.47

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Resolved

          Activity

            People

              rhn-engineering-rhusar Radoslav Husar
              mbabacek1@redhat.com Michal Karm
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: