An EJB on a WildFly server can't call another bean on a second server with remoting and SSL

I configured two WildFly server, one is the service provider (server), the other one the consumer (client). An EJB on the consumer server tries to call an EJB on the provider server and the remoting connection is secured by SSL. But it doesn't work. I can see the negotiation of the cipher suites and then the communication stops. I get a
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
on the provider side. A JavaSE client can call the service provider, but only when SSL_SARTTLS is not set. My current setup is:

subsystem undertow, default-server:

<https-listener name="defaults" socket-binding="https" security-realm="HttpsRealm"/>

subsystem ejb3:

<remote connector-ref="https-remoting-connector" thread-pool-name="default"/>

Output on the service provider side when the communication stops:

[stdout] (default I/O-3) Using SSLEngineImpl.
[stdout] (default I/O-3) Ignoring unavailable cipher suite: ...
*** 13 lines more ***
[stdout] (default I/O-3) Allow unsafe renegotiation: false
[stdout] (default I/O-3) Allow legacy hello messages: true
[stdout] (default I/O-3) Is initial handshake: true
[stdout] (default I/O-3) Is secure renegotiation: false
[stdout] (default I/O-3) default I/O-3, fatal error: 80: problem unwrapping net record
[stdout] (default I/O-3) javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
[stdout] (default I/O-3) default I/O-3, SEND TLSv1 ALERT:  fatal, description = internal_error
[stdout] (default I/O-3) default I/O-3, WRITE: TLSv1 Alert, length = 2
[stdout] (default I/O-3) default I/O-3, called closeOutbound()
[stdout] (default I/O-3) default I/O-3, closeOutboundInternal()
[stdout] (default I/O-3) [Raw write]: length = 7
[stdout] (default I/O-3) 0000: 15 03 01 00 02 02 50                               ......P

WildFly consumer config

subsystem remoting, outbound connections:

<remote-outbound-connection name="provider-one-connection" outbound-socket-binding-ref="provider-ejb" security-realm="ProviderOneRealm" protocol="https-remoting">
  <properties>
    <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
  </properties>
</remote-outbound-connection>

The realms contain the key- and truststores and all passwords are configured.

The reason for the communication problem comes from the following code.

The service consumer (or client) WildFly has a remote-outbound-connection in the remoting subsystem config which results in an org.jboss.as.remoting.RemoteOutboundConnectionService. There is a connect() method which contains the lines:

OptionMap.Builder builder = OptionMap.builder();
builder.addAll(this.connectionCreationOptions);
builder.set(SASL_POLICY_NOANONYMOUS, Boolean.FALSE);
builder.set(SASL_POLICY_NOPLAINTEXT, Boolean.FALSE);
builder.set(Options.SASL_DISALLOWED_MECHANISMS, Sequence.of(JBOSS_LOCAL_USER));
builder.set(Options.SSL_ENABLED, true);
builder.set(Options.SSL_STARTTLS, true);

return endpoint.connect(uri, builder.getMap(), callbackHandler, sslContext);

My configuration comes from this.connectionOptions and is overwritten by the defaults. The service consumer will open the connection with SSL_STARTTLS==true.

The service provider (or server) WildFly uses an https-listener from the undertow subsystem. Which generates an org.wildfly.extension.undertow.HttpsListenerService. Method startListening(XnioWorker worker,...) has the following code:

protected void startListening(XnioWorker worker, InetSocketAddress socketAddress, ChannelListener<AcceptingChannel<StreamConnection>> acceptListener) throws IOException {
    SSLContext sslContext = securityRealm.getValue().getSSLContext();
    Builder builder = OptionMap.builder().addAll(SERVER_OPTIONS);
    builder.set(UndertowOptions.MAX_ENTITY_SIZE, maxUploadSize);
    if (securityRealm.getValue().getSupportedAuthenticationMechanisms().contains(AuthMechanism.CLIENT_CERT)) {
        builder.set(SSL_CLIENT_AUTH_MODE, REQUESTED);
    }
    builder.set(Options.USE_DIRECT_BUFFERS,true);
    OptionMap combined = builder.getMap();

    XnioSsl xnioSsl = new JsseXnioSsl(worker.getXnio(), combined, sslContext);
    sslServer = xnioSsl.createSslConnectionServer(worker, socketAddress, (ChannelListener) acceptListener, combined);
    sslServer.resumeAccepts();

    UndertowLogger.ROOT_LOGGER.listenerStarted("HTTPS", getName(), socketAddress);
}

The OptionMap combined is used for the JsseXnioSsl and I can't see a way how to add my own configuration options. The result is that SSL_STARTTLS is undefined and in JsseXnioSsl has a method connectSsl which calls openSslConnection. There is an event handler build which calls:

new JsseSslStreamConnection(connection, JsseSslUtils.createSSLEngine(sslContext, optionMap, destination), socketBufferPool, applicationBufferPool, optionMap.get(Options.SSL_STARTTLS, false));

This sets the flag startTls of the constructor to false and the member variable tls in org.xnio.ssl.JsseSslStreamConnection to true. And when I understand it right, will then the consumer start unencrypted and will only switch to TLS when it is told todo so. But the service provider expectes, because of tls==true, an encrypted connection right from the beginning. And this generates the above error message/exception. I can reproduce this by using a JavaSE client to call the service provider. This works well when I don't set SSL_STARTTLS on the client side. But when I set it to true I get the same behavior like when the client is another WildFly instance.

tomazcerar said that on the consumer side in the connect() method the builder.addAll(...) should be moved to the end after setting the defaults. Thats the reason for this issue.

With this solution I can switch off SSL_STARTTLS on the consumer(client) side.

Another additional possibility could be to make the provider (server) side configurable that the hard coded options can be overwritten. But I don't know if this fits in the original design.