Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-12016

Outbound channel count will become divergent from actually open channels when remote service is not yet registered

    Details

    • Type: Bug
    • Status: Open (View Workflow)
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 15.0.1.Final
    • Fix Version/s: None
    • Component/s: Remoting
    • Labels:
      None

      Description

      Requesting the opening of an outbound channel via org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) for a service type that is not registered on the remote node results in a divergence between the number of channels actually opened and the internal counter.

      In the worst case this will result in an unusable connection to the remote node that will never recover and can only be fixed by restarting the application server from which the outbound channel should have been opened.

      In our case, the problem came up after restarting the remote application server, resulting in the following repeated exception:
      org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080
      at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83)
      at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
      at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
      at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
      at java.lang.Thread.run(Unknown Source)
      Suppressed: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080
      ... 7 more

      The following excerpt from checking the Service MBean to dump the channel status showed a maxed outbound channel counter but no open channels:
      Connection LOCAL-NODE/172.30.15.151:59765 <-> /172.30.15.150:8080
      Raw: org.xnio.ssl.JsseSslStreamConnection@31166dfc

      • Flags: supports-message-close auth-cap
      • 0 (max 40) inbound channels
      • 40 (max 40) outbound channels
      • Channels:
        Connection LOCAL-NODE/172.30.15.151:57690 <-> /172.30.15.152:8080

      Reproducing the problem with activated org.jboss.remoting traces showed, that the remote application server was refusing the channel open request (for the jboss.ejb service type):

      14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel 2d4e99c8: Unknown service name

      Meanwhile, the local application server was trying to open channels until the maximum of 40 channels was reached.

      The propes fix (as we did locally):

      When handling a service error response in org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel) properly decrement the internal counter by calling org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match the previously succeeded call to
      org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap).

      diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      index ceb0e32e..bccd046a 100644
      — a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      +++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
      @@ -424,6 +424,9 @@ final class RemoteReadListener implements ChannelListener<ConduitStreamSourceCha
      // invalid
      break;
      }
      + // match the handleOutboundChannelOpened() call in RemoteConnectionHandler.open(String, Result<Channel>, OptionMap)
      + handler.handleOutboundChannelClosed();
      + log.tracef("Properly decremented outbound channel counter after closing pending channel %08x.", channelId);
      String reason = new String(Buffers.take(buffer), StandardCharsets.UTF_8);
      pendingChannel.getResult().setException(new ServiceOpenException(reason));
      break;

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                flavia.rainone Flavia Rainone
                Reporter:
                petrjoac Joachim Petrich
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: