Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-11548

HTTP/2: WildFly worker's interoperability with Apache HTTP Server mod_cluster

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Critical
    • Resolution: Done
    • Affects Version/s: 7.1.0.ER1
    • Fix Version/s: 7.1.0.ER2
    • Component/s: mod_cluster
    • Labels:
      None
    • Target Release:
    • Affects Testing:
      Blocks Testing

      Description

      There are two configurations debated on this JIRA, the first seems to work O.K. with some concerns, the second one doesn't work at all .

      1. "ALPN hack", Oracle JDK8

      Hello, when you have this configuration of Apache HTTP Server 2.4.23 mod_cluster.conf and EAP 7.1 standalone-ha.xml and you start both servers, the worker correctly contacts the balancer and it is able to serve requests via HTTP/2.0. Although, I am mildly concerned about the irregular messages I can see during httpd <-> eap periodic 5s liveness verification (LBstatusRecalTime) with OPTIONS method and about connector's exceptions during sending MCMP messages. I haven't tried to let it run for days, but long-time impact is my main motivation for posting this. I parsed the log into corresponding blocks to illustrate situation on the balancer side and on the worker side.

      cap (wireshark)

      This is the recording of the initial communication worker -> balancer (registering message), balancer -> worker (response): JBEAP-11548_good.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

      Start

      Both servers started, no client requests whatsoever. The first block is the very first communication between servers.

      Worker Block 1

      05:33:02,544 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
      05:33:02,695 DEBUG [io.undertow.request] (default I/O-2) Using ALPN provider JDK8AlpnProvider for connector at /192.168.122.172:8443

      Balancer Block 1

      Full block

      Worker Block 2

      No exception, no log.

      Balancer Block 2

      Full block

      Worker Block 3

      05:33:23,144 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
          at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
          at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
          at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
          at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
          at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
          at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 3

      Full block

      Worker Block 4

      No exception, no log.

      Balancer Block 4

      Full block

      Worker Block 5

      No exception, no log.

      Balancer Block 5

      Full block

      Worker Block 6

      No exception, no log.

      Balancer Block 6

      Full block

      Worker Block 7

      No exception, no log.

      Balancer Block 7

      Full block

      Worker Block 8

      No exception, no log.

      Balancer Block 8

      Full block

      Worker Block 9

      No exception, no log.

      Balancer Block 9

      Full block

      Worker Block 10

      05:33:58,238 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

      Full block

      Balancer Block 10

      Full block

      Worker Block 11

      05:34:02,808 DEBUG [org.jboss.modcluster] (UndertowEventHandlerAdapter - 1) MODCLUSTER000009: Sending STATUS for default-server
      05:34:03,002 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
          at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
          at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
          at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
          at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
          at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
          at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 11

      Full block

      Worker Block 12

      No exception, no log.

      Balancer Block 12

      Full block

      Worker Block 13

      05:34:13,242 DEBUG [io.undertow.request] (default I/O-4) UT005013: An IOException occurred: java.nio.channels.ClosedChannelException
          at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:844)
          at io.undertow.protocols.ssl.SslConduit.doHandshake(SslConduit.java:647)
          at io.undertow.protocols.ssl.SslConduit.access$900(SslConduit.java:63)
          at io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1098)
          at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
          at org.xnio.nio.WorkerThread.run(WorkerThread.java:567)

      Balancer Block 13

      Full block

      Worker Block 14

      05:34:18,230 DEBUG [io.undertow.request.io] (default I/O-2) UT005013: An IOException occurred: java.io.IOException: Connection reset by peer

      Full block

      Balancer Block 14

      Full block

      Clients

      Worker is correctly registered:

      mod_cluster/1.3.5.Final
      Node jboss-eap-7.1 (https://192.168.122.172:8443):
      Balancer: qacluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 2,Ttl: 60000000,Status: OK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 99
      Virtual Host 1:
      Contexts:
        /clusterbench, Status: ENABLED Request: 0 Disable Stop
        /, Status: ENABLED Request: 0 Disable Stop
      Aliases:
        default-host
        localhost

      Curl

      Not related to EAP, nss Fedora 25 shenanigans with using blacklisted cipher suite: tls cipher ECDHE-RSA-AES256-SHA blacklisted by rfc7540

      Chrome

      Firefox

      2. ALPN via Wildfly OpenSSL + OpenSSL 1.0.2h + Oracle JDK8

      With the same Apache HTTP Server config as in the aforementioned case, i.e. mod_cluster.conf, and EAP 7.1 configured for Wildfly OpenSSL: standalone-ha.xml_openssl, the whole setup falls apart at the very beginning when worker contacts the balancer and then is unable to parse balancer's response. Noteworthy: If you put another EAP configured as an Undertow mod_cluster balancer in front of this worker, also with Wildfly OpenSSL, it works.

      cap (wireshark)

      This is the recording of the initial communication worker -> balancer (registering message), followed immediately by worker's crash during handshake, i.e. there is no balancer -> worker message: JBEAP-11548_bad.pcap, note port 8443 is the EAP's https connector and port 8747 is the Apache HTTP Server MCMP enabled, https enabled virtual host with both h2 and http/1.1 allowed, Protocols h2 http/1.1 and ProtocolsHonorOrder on.

      Balancer

      Everything appears cool, mod_manager.c(3104): manager_handler INFO OK, see the log snippet.

      Worker

      Falls flat on its face while processing balancer's response:

      06:16:24,031 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NumberFormatException: null
          at java.lang.Integer.parseInt(Integer.java:542)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:702)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
          at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454)
          at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          at org.jboss.threads.JBossThread.run(JBossThread.java:320)

      , see full log

      Result

      No worker is registered, integration is broken. The same worker setup with another Wildfly (EAP) Undertow mod_cluster balancer works.

      WDYT guys, Radoslav Husar, Stuart Douglas, Jean-Frederic Clere?

        Attachments

        1. JBEAP-11548_bad.pcap
          11 kB
          Michal Karm
        2. JBEAP-11548_good.pcap
          58 kB
          Michal Karm

          Issue Links

            Activity

              People

              Assignee:
              swd847 Stuart Douglas
              Reporter:
              mbabacek Michal Karm
              Tester:
              Michal Karm Michal Karm
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: