Outbound channel count will become divergent from actually open channels when remote service is not yet registered

Requesting the opening of an outbound channel via org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap) for a service type that is not registered on the remote node results in a divergence between the number of channels actually opened and the internal counter.

In the worst case this will result in an unusable connection to the remote node that will never recover and can only be fixed by restarting the application server from which the outbound channel should have been opened.

In our case, the problem came up after restarting the remote application server, resulting in the following repeated exception:
org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080
at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:83)
at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Unknown Source)
Suppressed: org.jboss.ejb.client.RequestSendFailedException: org.jboss.remoting3.ProtocolException: Too many channels open@remote+http://172.30.15.150:8080
... 7 more

The following excerpt from checking the Service MBean to dump the channel status showed a maxed outbound channel counter but no open channels:
Connection LOCAL-NODE/172.30.15.151:59765 <-> /172.30.15.150:8080
Raw: org.xnio.ssl.JsseSslStreamConnection@31166dfc

• Flags: supports-message-close auth-cap
• 0 (max 40) inbound channels
• 40 (max 40) outbound channels
• Channels:
  Connection LOCAL-NODE/172.30.15.151:57690 <-> /172.30.15.152:8080

Reproducing the problem with activated org.jboss.remoting traces showed, that the remote application server was refusing the channel open request (for the jboss.ejb service type):

14:00:26,672 TRACE [org.jboss.remoting.remote] (default I/O-4) Refusing service on channel 2d4e99c8: Unknown service name

Meanwhile, the local application server was trying to open channels until the maximum of 40 channels was reached.

The propes fix (as we did locally):

When handling a service error response in org.jboss.remoting3.remote.RemoteReadListener.handleEvent(ConduitStreamSourceChannel) properly decrement the internal counter by calling org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelClosed() to match the previously succeeded call to
org.jboss.remoting3.remote.RemoteConnectionHandler.handleOutboundChannelOpen() in org.jboss.remoting3.remote.RemoteConnectionHandler.open(String, Result<Channel>, OptionMap).

diff --git a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
index ceb0e32e..bccd046a 100644
— a/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
+++ b/src/main/java/org/jboss/remoting3/remote/RemoteReadListener.java
@@ -424,6 +424,9 @@ final class RemoteReadListener implements ChannelListener<ConduitStreamSourceCha
// invalid
break;
}
+ // match the handleOutboundChannelOpened() call in RemoteConnectionHandler.open(String, Result<Channel>, OptionMap)
+ handler.handleOutboundChannelClosed();
+ log.tracef("Properly decremented outbound channel counter after closing pending channel %08x.", channelId);
String reason = new String(Buffers.take(buffer), StandardCharsets.UTF_8);
pendingChannel.getResult().setException(new ServiceOpenException(reason));
break;