-
Bug
-
Resolution: Done-Errata
-
Normal
-
4.13, 4.12, 4.11, 4.10, 4.9, 4.14, 4.15, 4.16
This is a clone of issue OCPBUGS-36208. The following is the description of the original issue:
—
Description of problem
As a cluster-admin, I want to configure the router default connect timeout, so that HAProxy doesn't time out when connecting to a slow backend server, or so that I can set a shorter timeout to mitigate DoS attacks. This setting is controlled by the ROUTER_DEFAULT_CONNECT_TIMEOUT environment variable on the router deployment, which OpenShift router uses to set HAProxy's timeout connect setting.
This issue is being tracked as a defect rather than a new feature request because configurability of ROUTER_DEFAULT_CONNECT_TIMEOUT / timeout connect was an explicit requirement for RFE-403 (which was accepted) and NE-412 (which was completed and closed), but somehow the option got dropped during the design and implementation of the feature. This should have been caught during review of the pull request for the enhancement proposal, and the EP should have either included the option or justified its exclusion. As it stands, the feature is incomplete, and therefore this is a defect, not an RFE.
Version-Release number of selected component (if applicable)
4.16 through 4.9. The feature that was supposed to include this option shipped in OCP 4.9.
How reproducible
100%.
Steps to Reproduce
1. Check the supported router tuning options: oc explain IngressController.spec.tuningOptions
Actual results
There is no option to configure the server connect timeout:
% oc explain IngressController.spec.tuningOptions
KIND: IngressController
VERSION: operator.openshift.io/v1
RESOURCE: tuningOptions <Object>
DESCRIPTION:
tuningOptions defines parameters for adjusting the performance of ingress
controller pods. All fields are optional and will use their respective
defaults if not set. See specific tuningOptions fields for more details.
Setting fields within tuningOptions is generally not recommended. The
default values are suitable for most configurations.
FIELDS:
clientFinTimeout <string>
clientFinTimeout defines how long a connection will be held open while
waiting for the client response to the server/backend closing the
connection. If unset, the default timeout is 1s
clientTimeout <string>
clientTimeout defines how long a connection will be held open while waiting
for a client response. If unset, the default timeout is 30s
headerBufferBytes <integer>
headerBufferBytes describes how much memory should be reserved (in bytes)
for IngressController connection sessions. Note that this value must be at
least 16384 if HTTP/2 is enabled for the IngressController
(https://tools.ietf.org/html/rfc7540). If this field is empty, the
IngressController will use a default value of 32768 bytes. Setting this
field is generally not recommended as headerBufferBytes values that are too
small may break the IngressController and headerBufferBytes values that are
too large could cause the IngressController to use significantly more
memory than necessary.
headerBufferMaxRewriteBytes <integer>
headerBufferMaxRewriteBytes describes how much memory should be reserved
(in bytes) from headerBufferBytes for HTTP header rewriting and appending
for IngressController connection sessions. Note that incoming HTTP requests
will be limited to (headerBufferBytes - headerBufferMaxRewriteBytes) bytes,
meaning headerBufferBytes must be greater than headerBufferMaxRewriteBytes.
If this field is empty, the IngressController will use a default value of
8192 bytes. Setting this field is generally not recommended as
headerBufferMaxRewriteBytes values that are too small may break the
IngressController and headerBufferMaxRewriteBytes values that are too large
could cause the IngressController to use significantly more memory than
necessary.
healthCheckInterval <string>
healthCheckInterval defines how long the router waits between two
consecutive health checks on its configured backends. This value is applied
globally as a default for all routes, but may be overridden per-route by
the route annotation "router.openshift.io/haproxy.health.check.interval".
Expects an unsigned duration string of decimal numbers, each with optional
fraction and a unit suffix, eg "300ms", "1.5h" or "2h45m". Valid time units
are "ns", "us" (or "µs" U+00B5 or "μs" U+03BC), "ms", "s", "m", "h".
Setting this to less than 5s can cause excess traffic due to too frequent
TCP health checks and accompanying SYN packet storms. Alternatively,
setting this too high can result in increased latency, due to backend
servers that are no longer available, but haven't yet been detected as
such. An empty or zero healthCheckInterval means no opinion and
IngressController chooses a default, which is subject to change over time.
Currently the default healthCheckInterval value is 5s. Currently the
minimum allowed value is 1s and the maximum allowed value is 2147483647ms
(24.85 days). Both are subject to change over time.
maxConnections <integer>
maxConnections defines the maximum number of simultaneous connections that
can be established per HAProxy process. Increasing this value allows each
ingress controller pod to handle more connections but at the cost of
additional system resources being consumed. Permitted values are: empty, 0,
-1, and the range 2000-2000000. If this field is empty or 0, the
IngressController will use the default value of 50000, but the default is
subject to change in future releases. If the value is -1 then HAProxy will
dynamically compute a maximum value based on the available ulimits in the
running container. Selecting -1 (i.e., auto) will result in a large value
being computed (~520000 on OpenShift >=4.10 clusters) and therefore each
HAProxy process will incur significant memory usage compared to the current
default of 50000. Setting a value that is greater than the current
operating system limit will prevent the HAProxy process from starting. If
you choose a discrete value (e.g., 750000) and the router pod is migrated
to a new node, there's no guarantee that that new node has identical
ulimits configured. In such a scenario the pod would fail to start. If you
have nodes with different ulimits configured (e.g., different tuned
profiles) and you choose a discrete value then the guidance is to use -1
and let the value be computed dynamically at runtime. You can monitor
memory usage for router containers with the following metric:
'container_memory_working_set_bytes{container="router",namespace="openshift-ingress"}'.
You can monitor memory usage of individual HAProxy processes in router
containers with the following metric:
'container_memory_working_set_bytes{container="router",namespace="openshift-ingress"}/container_processes{container="router",namespace="openshift-ingress"}'.
reloadInterval <string>
reloadInterval defines the minimum interval at which the router is allowed
to reload to accept new changes. Increasing this value can prevent the
accumulation of HAProxy processes, depending on the scenario. Increasing
this interval can also lessen load imbalance on a backend's servers when
using the roundrobin balancing algorithm. Alternatively, decreasing this
value may decrease latency since updates to HAProxy's configuration can
take effect more quickly. The value must be a time duration value; see
<https://pkg.go.dev/time#ParseDuration>. Currently, the minimum value
allowed is 1s, and the maximum allowed value is 120s. Minimum and maximum
allowed values may change in future versions of OpenShift. Note that if a
duration outside of these bounds is provided, the value of reloadInterval
will be capped/floored and not rejected (e.g. a duration of over 120s will
be capped to 120s; the IngressController will not reject and replace this
disallowed value with the default). A zero value for reloadInterval tells
the IngressController to choose the default, which is currently 5s and
subject to change without notice. This field expects an unsigned duration
string of decimal numbers, each with optional fraction and a unit suffix,
e.g. "300ms", "1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"
U+00B5 or "μs" U+03BC), "ms", "s", "m", "h". Note: Setting a value
significantly larger than the default of 5s can cause latency in observing
updates to routes and their endpoints. HAProxy's configuration will be
reloaded less frequently, and newly created routes will not be served until
the subsequent reload.
serverFinTimeout <string>
serverFinTimeout defines how long a connection will be held open while
waiting for the server/backend response to the client closing the
connection. If unset, the default timeout is 1s
serverTimeout <string>
serverTimeout defines how long a connection will be held open while waiting
for a server/backend response. If unset, the default timeout is 30s
threadCount <integer>
threadCount defines the number of threads created per HAProxy process.
Creating more threads allows each ingress controller pod to handle more
connections, at the cost of more system resources being used. HAProxy
currently supports up to 64 threads. If this field is empty, the
IngressController will use the default value. The current default is 4
threads, but this may change in future releases. Setting this field is
generally not recommended. Increasing the number of HAProxy threads allows
ingress controller pods to utilize more CPU time under load, potentially
starving other pods if set too high. Reducing the number of threads may
cause the ingress controller to perform poorly.
tlsInspectDelay <string>
tlsInspectDelay defines how long the router can hold data to find a
matching route. Setting this too short can cause the router to fall back to
the default certificate for edge-terminated or reencrypt routes even when a
better matching certificate could be used. If unset, the default inspect
delay is 5s
tunnelTimeout <string>
tunnelTimeout defines how long a tunnel connection (including websockets)
will be held open while the tunnel is idle. If unset, the default timeout
is 1h
% oc version
Client Version: 4.13.0-0.ci-2022-11-11-144318
Kustomize Version: v4.5.7
Server Version: 4.12.0-0.nightly-2024-02-06-121927
Kubernetes Version: v1.25.16+6df2177
%
Expected results
spec.tuningOptions should have an option to configure the connect timeout. For example:
connectTimeout <string>
connectTimeout defines how long the router will wait for a response when
establishing a connection to a backend server. If unset, the default
timeout is 5s.
- clones
-
OCPBUGS-36208 [release-4.15] Allow configuring router default connect timeout on an ingress controller
-
- Closed
-
- is blocked by
-
OCPBUGS-36208 [release-4.15] Allow configuring router default connect timeout on an ingress controller
-
- Closed
-
- links to
-
RHBA-2024:4960
OpenShift Container Platform 4.14.z bug fix update