disconnected hostedcluster upgrade failed(4.16.17 to 4.17.5) and report CanaryChecksRepetitiveFailures

OCP Version: 4.18.10
MCE: 2.8.2
OpenShift Virtualization: 4.18.2

guest cluster
---
OCP Version: 4.16.17

upgrade hostedcluster from 4.16.17 to 4.17.5    

1. 
oc get HostedCluster jwang-hcp-demo -o json | jq -r '.spec.release.image="helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64"' | oc apply -f -     

2.
oc annotate hostedcluster -n clusters jwang-hcp-demo "hypershift.openshift.io/force-upgrade-to=helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64" --overwrite  

$ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress 
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
ingress   4.17.5    True        False         True       19h     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:...

$ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress -o json | jq .status.conditions[2] 
{
  "lastTransitionTime": "2025-06-04T03:10:42Z",
  "message": "The \"default\" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Get \"https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Bad Gateway (x1112 over 18h31m57s))",
  "reason": "IngressDegraded",
  "status": "True",
  "type": "Degraded"
}

From Hosted Control Plane Pod ingress-operator access https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg failed
$ oc exec -it -n clusters-jwang-hcp-demo ingress-operator-6cb6866f5b-2b2h5 -- curl -v https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg 
Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, availability-prober (init)
* Uses proxy env variable NO_PROXY == 'kube-apiserver'
* Uses proxy env variable HTTPS_PROXY == 'http://127.0.0.1:8090'
*   Trying 127.0.0.1:8090...
* Connected to 127.0.0.1 (127.0.0.1) port 8090 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443
> CONNECT canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443 HTTP/1.1
> Host: canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443
> User-Agent: curl/7.76.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 502 Bad Gateway
< Content-Type: text/plain
< Content-Length: 380
< 
* Received HTTP code 502 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 502 from proxy after CONNECT
command terminated with exit code 56

Testing the Konnectivity server with curl
$ sh /tmp/test-konnectivity.sh
work directory is: /tmp/tmp.kI5bhZJWGA
Forwarding from 127.0.0.1:8090 -> 8090
Forwarding from [::1]:8090 -> 8090
Handling connection for 8090
# HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
# TYPE aggregator_discovery_aggregation_count_total counter
aggregator_discovery_aggregation_count_total 0
...    

Hostedcluster upgrade success