SNO installation failing on Fedora 39: CanaryChecksSucceeding=False

    Installation of OpenShift SNO (different version from 4.12 and newer fail on my recently installed Fedora 39 host.  The following cluster operators are degraded: authentication, console ingress

    always

    1.Create cluster at https://console.redhat.com/openshift/assisted-installer/clusters/~new 2. download "full ISO image (with LVM)" 3. start installation

$ oc get co | awk 'NF > 6 {print $0}'
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.15.3    False       False         True       7m27s   OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.ocpa.ocp.internal/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
console                                    4.15.3    False       True          True       97m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.ocpa.ocp.internal): Get "https://console-openshift-console.apps.ocpa.ocp.internal": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
ingress                                    4.15.3    True        False         True       3h48m   The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing)
    

    working cluster

    https://access.redhat.com/solutions/5891131 did not resolve the issue

The following is the output of the commands in https://access.redhat.com/solutions/5891131
 
$ SVC_IP=$(oc get svc -n openshift-ingress-canary -ojsonpath={..clusterIP})
$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i – curl http://${SVC_IP}:8080 -s -D - ; done
HTTP/1.1 200 OK
X-Request-Port: 8080
Date: Thu, 28 Mar 2024 20:25:43 GMT
Content-Length: 22
Content-Type: text/plain; charset=utf-8
Healthcheck requested
$ ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i – curl http://${ROUTE} -sS -k -D - ; done
HTTP/1.1 302 Found
content-length: 0
location: https://canary-openshift-ingress-canary.apps.ocpa.ocp.internal/
cache-control: no-cache
$ ROUTE=$(oc get route -n openshift-ingress-canary -ojsonpath={..host})
$ for i in `oc get po -n openshift-ingress-operator | grep -v NAME| awk '{print $1}' ` ; do oc exec -n openshift-ingress-operator -c ingress-operator $i – dig ${ROUTE} +nocmd +noall +answer ; done
canary-openshift-ingress-canary.apps.ocpa.ocp.internal.    5 IN A 192.168.122.150
$ oc get pods -o wide -n openshift-ingress
NAME                              READY   STATUS    RESTARTS   AGE   IP                NODE   NOMINATED NODE   READINESS GATES
router-default-5b4bf785f9-ln9gr   1/1     Running   0          25m   192.168.122.150   ocpa   <none>           <none>
$ curl -kv --resolve ${ROUTE}:80:192.168.122.150 http://$
{ROUTE}
* processing: http://canary-openshift-ingress-canary.apps.ocpa.ocp.internal
* Added canary-openshift-ingress-canary.apps.ocpa.ocp.internal:80:192.168.122.150 to DNS cache
* Hostname canary-openshift-ingress-canary.apps.ocpa.ocp.internal was found in DNS cache
*   Trying 192.168.122.150:80...
* Connected to canary-openshift-ingress-canary.apps.ocpa.ocp.internal (192.168.122.150) port 80
> GET / HTTP/1.1
> Host: canary-openshift-ingress-canary.apps.ocpa.ocp.internal
> User-Agent: curl/8.2.1
> Accept: /
> 
< HTTP/1.1 302 Found
< content-length: 0
< location: https://canary-openshift-ingress-canary.apps.ocpa.ocp.internal/
< cache-control: no-cache
< 
* Connection #0 to host canary-openshift-ingress-canary.apps.ocpa.ocp.internal left intact
$ curl -kv --resolve ${ROUTE}:443:192.168.122.150 https://${ROUTE}

processing: https://canary-openshift-ingress-canary.apps.ocpa.ocp.internal
Added canary-openshift-ingress-canary.apps.ocpa.ocp.internal:443:192.168.122.150 to DNS cache
Hostname canary-openshift-ingress-canary.apps.ocpa.ocp.internal was found in DNS cache
  Trying 192.168.122.150:443...
Connected to canary-openshift-ingress-canary.apps.ocpa.ocp.internal (192.168.122.150) port 443
ALPN: offers h2,http/1.1
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
ALPN: server did not agree on a protocol. Uses default.
Server certificate:
 subject: CN=*.apps.ocpa.ocp.internal
 start date: Mar 28 16:27:46 2024 GMT
 expire date: Mar 28 16:27:47 2026 GMT
 issuer: CN=ingress-operator@1711643266
 SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
using HTTP/1.x
> GET / HTTP/1.1
> Host: canary-openshift-ingress-canary.apps.ocpa.ocp.internal
> User-Agent: curl/8.2.1
> Accept: /
> 
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< x-request-port: 8080
< date: Thu, 28 Mar 2024 19:53:17 GMT
< content-length: 22
< content-type: text/plain; charset=utf-8
< set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=b0cb1ae85d5d1fab68608cecda8bc167; path=/; HttpOnly; Secure; SameSite=None
< 
Healthcheck requested
Connection #0 to host canary-openshift-ingress-canary.apps.ocpa.ocp.internal left intact