Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-8753

OpenSSL via Wildfly-OpenSSL crashes JVM when mod_cluster used

XMLWordPrintable

    • User Experience

      Previously I thought it happens just on server shutdown...

      Worker node with this mod_cluster and Elytron configuration standalone-ha.xml keeps complaining on being unable to agree on a cipher suite with the Undertow balancer, whose Elytron configuration is identical, i.e.

      <server-ssl-contexts>
          <server-ssl-context name="serverSSLContext" providers="openssl" need-client-auth="false" key-managers="keyManager"  protocols="TLSv1.2" trust-managers="trustManager"/>
      </server-ssl-contexts>
      <client-ssl-contexts>
          <client-ssl-context name="clientSSLContext" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/>
      </client-ssl-contexts>

      see:

      15:19:25,492 FINE  [org.wildfly.openssl.OpenSSLEngine] (UndertowEventHandlerAdapter - 1) The version of SSL in use does not support cipher ordering
      15:19:25,541 ERROR [org.jboss.mod_cluster.undertow] (UndertowEventHandlerAdapter - 1) null: java.lang.NullPointerException
          at org.wildfly.openssl.CipherSuiteConverter.toJava(CipherSuiteConverter.java:284)
          at org.wildfly.openssl.OpenSSLEngine.toJavaCipherSuite(OpenSSLEngine.java:1021)
          at org.wildfly.openssl.OpenSSlSession.initCipherSuite(OpenSSlSession.java:305)
          at org.wildfly.openssl.OpenSSlSession.initialised(OpenSSlSession.java:296)
          at org.wildfly.openssl.OpenSSLSessionContext.clientSessionCreated(OpenSSLSessionContext.java:114)
          at org.wildfly.openssl.OpenSSLClientSessionContext.storeClientSideSession(OpenSSLClientSessionContext.java:92)
          at org.wildfly.openssl.OpenSSLEngine.handshakeFinished(OpenSSLEngine.java:940)
          at org.wildfly.openssl.OpenSSLEngine.getHandshakeStatus(OpenSSLEngine.java:989)
          at org.wildfly.openssl.OpenSSLEngine.unwrap(OpenSSLEngine.java:606)
          at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
          at org.wildfly.openssl.OpenSSLSocket.runHandshake(OpenSSLSocket.java:316)
          at org.wildfly.openssl.OpenSSLSocket.write(OpenSSLSocket.java:462)
          at org.wildfly.openssl.OpenSSLOutputStream.write(OpenSSLOutputStream.java:46)
          at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
          at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
          at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
          at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
          at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
          at java.io.BufferedWriter.flush(BufferedWriter.java:254)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:526)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:387)
          at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:365)
          at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:454)
          at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.run(UndertowEventHandlerAdapter.java:169)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
          at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      

      Full log: server.log.
      I might or might not have it misconfigured, but what definitely should not happen is that when I try to shut the server down:

      INFO  [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
      

      it crashes JVM (doublefree?):

      # JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [libcrypto.so.1.0.2h+0x117a64]  sk_pop_free+0x24
      #
      

      Core: hs_err_pid12281.log

      Now, with this configuration, it happens pretty much immediately after the worker tries to register with the balancer:

      Elytron

      <server-ssl-contexts>
          <server-ssl-context name="serverSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" need-client-auth="false" key-managers="keyManager"  protocols="TLSv1.2" trust-managers="trustManager"/>
      </server-ssl-contexts>
      <client-ssl-contexts>
          <client-ssl-context name="clientSSLContext" cipher-suite-filter="TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" providers="openssl" key-managers="clientKeyManager" protocols="TLSv1.2" trust-managers="trustManager"/>
      </client-ssl-contexts>
      

      Undertow

      <server name="default-server">
          <http-listener name="default" socket-binding="http" redirect-socket="https" enable-http2="true"/>
          <https-listener name="https" socket-binding="https" enable-http2="true" ssl-context="serverSSLContext"/>
          <host name="default-host" alias="localhost">
              <location name="/" handler="welcome-content"/>
              <access-log/>
              <filter-ref name="server-header"/>
              <filter-ref name="x-powered-by-header"/>
          </host>
      </server>
      

      mod_cluster

      <subsystem xmlns="urn:jboss:domain:modcluster:3.0">
          <mod-cluster-config advertise-socket="modcluster" connector="https" ssl-context="clientSSLContext">
              <dynamic-load-provider>
                  <load-metric type="cpu"/>
              </dynamic-load-provider>
          </mod-cluster-config>
      </subsystem>
      

      See new crashes_on_mod_cluster_connection.server.log and new_hs_err_pid13002.log.

      I must admit I am somewhat getting into low spirits with this whole client <--H/2-->mod_cluster balancer<--H/2-->mod_cluster worker and "just use Elytron" situation. Would it be possible for you to actually give it a try? Take 1 Wildfly and Undertow mod_cluster balancer, set it up with H/2 via Elytron, take 1 Wildfly node as H/2 mod_cluster worker configured with Elytron, start it...

        1. crashes_on_mod_cluster_connection.server.log
          300 kB
        2. hs_err_pid12281.log
          128 kB
        3. new_hs_err_pid13002.log
          144 kB
        4. server.log
          941 kB
        5. standalone-ha.xml
          33 kB

            sdouglas1@redhat.com Stuart Douglas
            mbabacek1@redhat.com Karm Karm
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: