Details
-
Feature Request
-
Resolution: Unresolved
-
Major
-
None
-
2.0.26.Final
-
None
Description
This is a bug that manifests in a number of different ways, but always when connecting via outbound websocket to a public Internet host protected by Cloudflare (identifiable by "Server: Cloudflare" response header):
- When connecting to a wss:// URL with a short request timeout (~10 seconds), it throws an IOException "UT003035: Request timed out".
- When connecting to a wss:// URL with a long request timeout (~30 seconds), the socket can connect and send the first message, but will throw a java.nio.channels.ClosedChannelException when the first message comes back from the server.
- When connecting to some ws:// URLs (eg. Poloniex), it returns a 429 Too Many Requests status code.
- When connecting to other ws:// URLs (eg. Coinbase), it follows several 301 Moved Permanently redirects and then throws an IOException "UT000145: Too many redirects". (The redirect loop is one way Cloudflare traps scrapers.)
I did a tcpdump on the output and compared it to a Chrome connection from the same IP, same machine, which succeeded:
GET / HTTP/1.1 Sec-WebSocket-Key: <key> Connection: upgrade Sec-WebSocket-Version: 13 Host: ws-feed.pro.coinbase.com:80 Upgrade: websocket
GET wss://ws-feed.pro.coinbase.com/ HTTP/1.1 Host: ws-feed.pro.coinbase.com Connection: Upgrade Pragma: no-cache Cache-Control: no-cache Upgrade: websocket Origin: https://pro.coinbase.com Sec-WebSocket-Version: 13 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36 Accept-Encoding: gzip, deflate, br Accept-Language: en-US,en;q=0.9 Sec-WebSocket-Key: <key> Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
There are some differences in Accept/Cache/Origin headers that are probably inconsequential, but the key difference is a lack of a User-Agent header. Cloudflare flags almost all traffic without a User-Agent as malicious, and other sites' DDOS protection may also flag it as suspicious.
I dug into the source code for ServerWebSocketContainer and associated classes. The relevant headers are set in io.undertow.websockets.client. WebSocket13ClientHandshake.createHeaders, which sets the headers observed above. AFAICT, there's no way to set additional headers on the initial HTTP request that opens the connection. My preference would be a configuration option (either on ClientEndpointConfig.getUserProperties() or on the OptionMap to construct the ServerWebSocketContainer) that would allow me to set these headers, and in particular a custom user agent. Failing that, I'd settle for Undertow to identify itself with a unique user-agent header, which at least would make it functional with Cloudflare-protected sites.