Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-12016

Outbound channel count will become divergent from actually open channels when remote service is not yet registered


    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 15.0.1.Final
    • Fix Version/s: None
    • Component/s: Remoting
    • Labels:


      Requesting the opening of an outbound channel via org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) for a service type that is not registered on the remote node results in a divergence between the number of channels actually opened and the internal counter.

      In the worst case this will result in an unusable connection to the remote node that will never recover and can only be fixed by restarting the application server from which the outbound channel should have been opened.

      In our case, the problem came up after restarting the remote application server, resulting in the following repeated exception:
      org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+
      at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83)
      at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
      at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
      at java.lang.Thread.run(Unknown Source)
      Suppressed: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+
      ... 7 more

      The following excerpt from checking the Service MBean to dump the channel status showed a maxed outbound channel counter but no open channels:
      Connection LOCAL-NODE/ <-> /
      Raw: org.xnio.ssl.JsseSslStreamConnection@31166dfc

      • Flags: supports-message-close auth-cap
      • 0 (max 40) inbound channels
      • 40 (max 40) outbound channels
      • Channels:
        Connection LOCAL-NODE/ <-> /

      Reproducing the problem with activated org.jboss.remoting traces showed, that the remote application server was refusing the channel open request (for the jboss.ejb service type):

      14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel 2d4e99c8: Unknown service name

      Meanwhile, the local application server was trying to open channels until the maximum of 40 channels was reached.

      The propes fix (as we did locally):

      When handling a service error response in org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel) properly decrement the internal counter by calling org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match the previously succeeded call to
      org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap).

      diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      index ceb0e32e..bccd046a 100644
      — a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      +++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      @@ -424,6 +424,9 @@ final class RemoteReadListener implements ChannelListener<ConduitStreamSourceCha
      // invalid
      + // match the handleOutboundChannelOpened() call in RemoteConnectionHandler.open(String, Result<Channel>, OptionMap)
      + handler.handleOutboundChannelClosed();
      + log.tracef("Properly decremented outbound channel counter after closing pending channel %08x.", channelId);
      String reason = new String(Buffers.take(buffer), StandardCharsets.UTF_8);
      pendingChannel.getResult().setException(new ServiceOpenException(reason));

        Gliffy Diagrams




              • Assignee:
                flavia.rainone Flavia Rainone
                petrjoac Joachim Petrich
              • Votes:
                0 Vote for this issue
                1 Start watching this issue


                • Created: