[#5221] Idle timeout of outbound broker connection may not trigger reconnection

https://github.com/EnMasseProject/enmasse/issues/5221

Internally AMQ Online uses the broker plugin to establish connections from the broker to the route network. It configures an idle timeout as part of the outbound open performative in order to achieve connectuvity robustness.

If the broker detects an idle timeout the broker should close the connection and proceed to reestablish a new replacement connection.  However, there is a defect. For the reconnection to occur, the peer (router) needs to shutdown the socket.  If this shutdown does not occur or is not detected (as might be the case under unusual network connections), the connection remains unmade.

Here's a trace where the peer responds to the Close by replying Close and closing shutting down the socket.

2020-12-23 12:44:24,543 INFO [org.apache.activemq.artemis.integration.amqp.ProtonClientConnectionManager] Connection localhost/127.0.0.1:15672 created
[322200002:0] -> AMQP
[322200002:0] -> Open{ containerId='foo', hostname='null', maxFrameSize=131072, channelMax=65535, idleTimeOut=8000, outgoingLocales=null, incomingLocales=null, offeredCapabilities=null, desiredCapabilities=null, properties={product=apache-activemq-artemis, version=2.13.0, qd.route-container-group=cluster}}
[322200002:0] <- AMQP
[322200002:0] <- Open{ containerId='test', hostname='null', maxFrameSize=4294967295, channelMax=65535, idleTimeOut=null, outgoingLocales=null, incomingLocales=null, offeredCapabilities=null, desiredCapabilities=null, properties=null}[322200002:0] -> Close{error=Error{condition=amqp:resource-limit-exceeded, description='local-idle-timeout expired', info=null}}
2020-12-23 12:44:40,544 INFO [org.apache.activemq.artemis.integration.amqp.ProtonClientConnectionManager] Connection localhost/127.0.0.1:15672 destroyed
2020-12-23 12:44:40,544 INFO [org.apache.activemq.artemis.integration.amqp] connectionDestroyed for connector amqp-connector

and for comparison, one where the peer consumes the close but fails to respond or shutdown the connection.

2020-12-23 12:47:07,610 INFO [org.apache.activemq.artemis.integration.amqp] Starting connector amqp-connector
2020-12-23 12:47:07,611 INFO [org.apache.activemq.artemis.integration.amqp] connectionCreated for connector amqp-connector
2020-12-23 12:47:07,611 INFO [org.apache.activemq.artemis.integration.amqp.ProtonClientConnectionManager] Connection localhost/127.0.0.1:15672 created
[1421049339:0] -> AMQP
[1421049339:0] -> Open{ containerId='foo', hostname='null', maxFrameSize=131072, channelMax=65535, idleTimeOut=8000, outgoingLocales=null, incomingLocales=null, offeredCapabilities=null, desiredCapabilities=null, properties={product=apache-activemq-artemis, version=2.13.0, qd.route-container-group=cluster}}
[1421049339:0] <- AMQP
[1421049339:0] <- Open{ containerId='test', hostname='null', maxFrameSize=4294967295, channelMax=65535, idleTimeOut=null, outgoingLocales=null, incomingLocales=null, offeredCapabilities=null, desiredCapabilities=null, properties=null}
[1421049339:0] -> Close{error=Error{condition=amqp:resource-limit-exceeded, description='local-idle-timeout expired', info=null}}

This defect does not affect inbound AMQP connections.