-
Bug
-
Resolution: Done-Errata
-
Major
-
4.14
Description of problem:
Due to HTTP/2 Connection Coalescing (https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/), routes which use the same certificate can present unexplained 503 errors when attempting to access an HTTP/2 enabled ingress controller.
It appears that HAProxy supports the ability to force HTTP 1.1 on a route-by-route basis, but our Ingress Controller does not expose that option.
This is especially problematic for component routes because generally speaking, customers use a wildcard or SAN to deploy custom component routes (console, OAuth, downloads), but with HTTP/2, this does not work properly.
To address this issue, we're proposing the creation of an annotation haproxy.router.openshift.io/http2-disable, which will allow the disabling of HTTP/2 on a route-by-route basis, or smarter logic built into our Ingress operator to handle this situation.
Version-Release number of selected component (if applicable):
OpenShift 4.14
How reproducible:
Serve routes to applications in Openshift. Observe the routes through a HTTP/2 enabled client. Notice that http/2 client connections are broken (returns 503 on second connection when using same certificates across a mix of re-encrypt and passthrough routes)
Steps to Reproduce:
(see above notes)
Actual results:
503 error
Expected results:
no error
Additional info:
- is triggering
-
NE-2005 Corrective Measure for OCPBUGS-29373: http/2 connection coalescing component routing broken with single certificate
-
- To Do
-
- links to
-
RHEA-2024:0041
OpenShift Container Platform 4.16.z bug fix update
[OCPBUGS-29373] http/2 connection coalescing component routing broken with single certificate
pre-merge test done, PR was included in night build 4.16.0-0.nightly-2024-05-04-214435, bot didn't move to verified, marked it verified manually, thanks
OpenShift Jira Bot
added a comment - Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi mmasters1@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. mmasters1@redhat.com I can reproduce this issue now, I just followed your steps but one more thing is importing the ca.crt to Chrome as a trusted Authority then access the console route, it will be redirected to auth and I can see "Application is not available".
hkemp mentioned that "In my example, I patch ingress object with the cluster's own primary-cert-bundle-secret, so that we can demonstrate the issue without having to have Chrome trust any self-signed certificates", I think if without importing the self-signed certificate then we need to accept the cert warning to proceed, that might make it cannot expose the 503 issue.
Hi mmasters1@redhat.com, I've managed to reproduce this on a stock OSD/ROSA cluster. I've followed your steps above, except for some minor differences. In your example, you patch the console route's domain with a custom certificate you generate using OpenSSL.
In my example, I patch ingress object with the cluster's own primary-cert-bundle-secret, so that we can demonstrate the issue without having to have Chrome trust any self-signed certificates (which in my experience, makes it hard to demonstrate the issue, since all certificate errors will be ignored anyway).
What I did:
Verify the default IC has http/2 enabled:
oc -n openshift-ingress-operator get ingresscontroller default -oyaml
apiVersion: operator.openshift.io/v1 kind: IngressController metadata: annotations: Owner: cluster-ingress-operator ingress.operator.openshift.io/auto-delete-load-balancer: "true" creationTimestamp: "2024-03-11T23:10:37Z" .....
Patch ingress object with primary cert bundle secret (on the console and OAuth routes, not just the console):
oc patch ingresses.config.openshift.io/cluster --type=merge --patch=" spec: componentRoutes: - hostname: console-openshift-console.$ingress_domain name: console namespace: openshift-console servingCertKeyPairSecret: name: hkemp-ing2-primary-cert-bundle-secret - hostname: oauth-openshift.$ingress_domain name: oauth-openshift namespace: openshift-authentication servingCertKeyPairSecret: name: hkemp-ing2-primary-cert-bundle-secret "
Verify alpn/http2 is enabled in HAProxy cert config map:
oc -n openshift-ingress rsh -c router deploy/router-default cat cert_config.map
/var/lib/haproxy/router/certs/openshift-console:console.pem [alpn h2,http/1.1] console-openshift-console.apps.hkemp-ing2.xu3j.s1.devshift.org
We can then see on the screencast I've attached below that the request to the console route is redirected to the Oauth route without opening a new TCP connection. Is this enough information? I am happy to set up a sync meet to discuss this further.
P.S: I find it really strange that you were unable to reproduce it at all with the OAuth route (coming back to the following quote):
> This did not help, and anyway it should be unnecessary as (1) the authentication operator configures OAuth to use the default certificate anyway and (2) the OAuth route uses TLS passthrough so it doesn't matter whether HAProxy allows HTTP/2 or not.
In the steps in my previous comment, I forgot to mention one step that must be done between steps 3 and 6:
oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch '{"spec":{"defaultCertificate":{"name":"custom-cert"}}}'
oc -n openshift-ingress create secret tls custom-cert --cert=tls.crt --key=tls.key
I actually did perform the step in my testing, or else step 6 would have shown the discrepancy. I only omitted the step in my write-up.
Besides that, I also tested with an NLB instead of the default classic ELB, just in case the ELB type somehow affects whether the connection gets re-used:
oc replace --force --filename=- <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
annotations:
ingress.operator.openshift.io/default-enable-http2: "true"
name: default
namespace: openshift-ingress-operator
spec:
defaultCertificate:
name: custom-cert
endpointPublishingStrategy:
type: LoadBalancerService
loadBalancer:
providerParameters:
type: AWS
aws:
type: NLB
scope: External
replicas: 2
EOF
I was still unable to reproduce the issue even with an NLB.
I still have not been able to come up with a reproducer. Following is what I have tried so far.
First, based on the video, I did the following:
1. Launch an OpenShift 4.14 cluster (not HyperShift, which runs OAuth on the management cluster rather than on the guest cluster), using Cluster Bot.
% oc version Client Version: 4.13.0-0.ci-2022-11-11-144318 Kustomize Version: v4.5.7 Server Version: 4.14.0-0.nightly-2024-02-22-202606 Kubernetes Version: v1.27.10+c79e5e2
2. Enable HTTP/2 on the default IngressController:
oc -n openshift-ingress-operator annotate ingresscontrollers/default ingress.operator.openshift.io/default-enable-http2=true
3. Generate a wildcard certificate that has SANs for the console and OAuth routes:
ingress_domain=$(oc get ingresses.config/cluster -o go-template --template='{{.spec.domain}}')
openssl req -x509 -newkey rsa:2048 -days 365 -keyout ca.key -out ca.crt -nodes -subj '/CN=example-ca' &&
openssl req -newkey rsa:2048 -nodes -keyout tls.key -out tls.csr -subj "/CN=*.$ingress_domain" &&
cat > tls.cfg <<EOF &&
[san]
subjectAltName = @alt_names
[alt_names]
DNS.1 = console-openshift-console.$ingress_domain
DNS.2 = oauth-openshift.$ingress_domain
EOF
openssl x509 -req -days 365 -in tls.csr -CA ca.crt -CAcreateserial -CAkey ca.key -out tls.crt -extensions san -extfile tls.cfg
4. Create a secret in the "openshift-config" namespace with this certificate and key:
oc -n openshift-config create secret tls custom-cert --cert=tls.crt --key=tls.key
5. Patch console component route to use this secret:
oc patch ingresses.config.openshift.io/cluster --type=merge --patch="
spec:
componentRoutes:
- hostname: console-openshift-console.$ingress_domain
name: console
namespace: openshift-console
servingCertKeyPairSecret:
name: custom-cert
"
6. Verify that the console route has HTTP/2 enabled and that the console certificate, OAuth certificate, and default certificate are the same:
% oc -n openshift-ingress rsh -c router deploy/router-default cat cert_config.map /var/lib/haproxy/router/certs/openshift-console:console.pem [alpn h2,http/1.1] console-openshift-console.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com % oc -n openshift-console get routes/console ; oc -n openshift-authentication get routes/oauth-openshift NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD console console-openshift-console.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com console https reencrypt/Redirect None NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD oauth-openshift oauth-openshift.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com oauth-openshift 6443 passthrough/Redirect None % openssl s_client -connect console-openshift-console.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com:443 </dev/null 2>/dev/null | openssl x509 -sha1 -in /dev/stdin -noout -fingerprint SHA1 Fingerprint=7E:AF:B5:E5:07:25:62:A4:82:C1:84:C4:AA:73:3C:25:70:23:C8:F4 % openssl s_client -connect oauth-openshift.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com:443 </dev/null 2>/dev/null | openssl x509 -sha1 -in /dev/stdin -noout -fingerprint SHA1 Fingerprint=7E:AF:B5:E5:07:25:62:A4:82:C1:84:C4:AA:73:3C:25:70:23:C8:F4 % openssl s_client -connect foo.apps.ci-ln-chtc9ck-76ef8.origin-ci-int-aws.dev.rhcloud.com:443 </dev/null 2>/dev/null | openssl x509 -sha1 -in /dev/stdin -noout -fingerprint SHA1 Fingerprint=7E:AF:B5:E5:07:25:62:A4:82:C1:84:C4:AA:73:3C:25:70:23:C8:F4
7. Go to Chrome, open the Developer Tools, select the Network tab, enable "Preserve log", and enable the "Url", "Protocol", and "Connection ID" columns.
Note: I tested using Chrome 120:
% rpm -qf =google-chrome google-chrome-stable-120.0.6099.199-1.x86_64
8. Browse to the console and try to log in using OAuth.
What I observed was that Chrome continued to use HTTP/1.1, even though the HAProxy configuration clearly shows that it should allow the browser to negotiate HTTP/2 using ALPN (per the cert_config.map entry for the route: [alpn h2,http/1.1]).
Next, I tried simply putting the default, operator-generated default certificate as the console route's certificate, which in theory should suffice to enable HTTP/2 connection coalescing for the console and OAuth routes:
oc -n openshift-ingress-operator patch ingresscontrollers/default --type=merge --patch '{"spec":{"defaultCertificate":null}}'
oc -n openshift-config create secret tls console-route-cert --cert=<(oc -n openshift-ingress get secrets/router-certs-default -o go-template --template='{{index .data "tls.crt"|base64decode}}') --key=<(oc -n openshift-ingress get secrets/router-certs-default -o go-template --template='{{index .data "tls.key"|base64decode}}')
oc patch ingresses.config.openshift.io/cluster --type=merge --patch="
spec:
componentRoutes:
- hostname: console-openshift-console.$ingress_domain
name: console
namespace: openshift-console
servingCertKeyPairSecret:
name: console-route-cert
"
This had the effect that the console allowed HTTP/2 connections, but the OAuth route still used HTTP/1.1.
I also tried configuring the OAuth route using componentRoutes:
oc patch ingresses.config.openshift.io/cluster --type=merge --patch="
spec:
componentRoutes:
- hostname: console-openshift-console.$ingress_domain
name: console
namespace: openshift-console
servingCertKeyPairSecret:
name: custom-cert
- hostname: oauth-openshift.$ingress_domain
name: oauth-openshift
namespace: openshift-authentication
servingCertKeyPairSecret:
name: custom-cert
"
This did not help, and anyway it should be unnecessary as (1) the authentication operator configures OAuth to use the default certificate anyway and (2) the OAuth route uses TLS passthrough so it doesn't matter whether HAProxy allows HTTP/2 or not.
I had intended to update the Jira ticket description with steps to reproduce the issue, in order to enable engineering to work on the issue (and eventually to enable QE to verify a fix). However, since I am still failing to reproduce the HTTP/2 connection coalescing issue, I hesitate to update the steps to reproduce it just yet.
Please let me know if you are able to spot an error in what I am doing.
Also, it anyone who manages to reproduce the issue could tell me the exact browser and version of that browser that is required to reproduce the issue, that would be very helpful!
Hey Hector - we need the info I asked for. Sadly, we don't really have the time to run through this video and try to type up everything from the video.
Hey Candace - see here for a reproducer in this PMR video:
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:0041