-
Bug
-
Resolution: Done
-
Major
-
2.0.16.Final
-
None
-
None
We encountered and issue with undertow in combination with temporarily unresolvable hostname. Undertow was started up and used before the DNS resolution became possible.
Our analysis show, that UnresolvedAddressException seems not beeing properly handled in Undertow 2.0.16. We found the following code locations ProxyConnectionPool (line 269 and following):
private void openConnection(final HttpServerExchange exchange, final ProxyCallback<ProxyConnection> callback, final HostThreadData data, final boolean exclusive) { if (!exclusive) { data.connections++; } client.connect(new ClientCallback<ClientConnection>() { @Override public void completed(final ClientConnection result) { openConnections.incrementAndGet(); final ConnectionHolder connectionHolder = new ConnectionHolder(result); if (!exclusive) { result.getCloseSetter().set(new ChannelListener<ClientConnection>() { @Override public void handleEvent(ClientConnection channel) { handleClosedConnection(data, connectionHolder); } }); } connectionReady(connectionHolder, callback, exchange, exclusive); } @Override public void failed(IOException e) { if (!exclusive) { data.connections--; } UndertowLogger.REQUEST_LOGGER.debug("Failed to connect", e); if (!connectionPoolManager.handleError()) { redistributeQueued(getData()); scheduleFailedHostRetry(exchange); } callback.failed(exchange); } }, bindAddress, getUri(), exchange.getIoThread(), ssl, exchange.getConnection().getByteBufferPool(), options); }
Obviously the counter data.connections (= number of connections to the backend per IO-Thread) gets incremented on open connection attempts and should be decremented in case of a failure in the failed(IOException e)-handler. However this seems not to happen in case of an UnresolvedAddressException (because it is not an IOException?).
Therefore the counter data.connections remains incremented, allthough no connection could have been established. And if data.connections reaches maxConnections, all further requests get rejected with a 503 response. No further attempts to resolve the hostname will be made, connection pool seems not having any way to recover. ProxyConnectionPool, line 526:
} else if (exclusive || data.connections < maxConnections) { openConnection(exchange, callback, data, exclusive); } else { // Reject the request directly if we reached the max request queue size if (data.awaitingConnections.size() >= connectionPoolManager.getMaxQueueSize()) { callback.queuedRequestFailed(exchange); return; }
—
Please see the provided example ch.codeblock.undertow.ReverseProxyServer in this GIT repo: https://github.com/gexclaude/undertow-unresolvablehost
And the test run outpus:
- unresolvable-host.md
- unresolvable-host-at-start-then-resolvable.md
