-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
4.16, 4.17
-
None
-
Quality / Stability / Reliability
-
False
-
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
disconnected hostedcluster upgrade failed(4.16.17 to 4.17.5) and report CanaryChecksRepetitiveFailures
Version-Release number of selected component (if applicable):
OCP Version: 4.18.10 MCE: 2.8.2 OpenShift Virtualization: 4.18.2 guest cluster --- OCP Version: 4.16.17
How reproducible:
upgrade hostedcluster from 4.16.17 to 4.17.5
Steps to Reproduce:
1. oc get HostedCluster jwang-hcp-demo -o json | jq -r '.spec.release.image="helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64"' | oc apply -f - 2. oc annotate hostedcluster -n clusters jwang-hcp-demo "hypershift.openshift.io/force-upgrade-to=helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64" --overwrite
Actual results:
$ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE ingress 4.17.5 True False True 19h The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:... $ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress -o json | jq .status.conditions[2] { "lastTransitionTime": "2025-06-04T03:10:42Z", "message": "The \"default\" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Get \"https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Bad Gateway (x1112 over 18h31m57s))", "reason": "IngressDegraded", "status": "True", "type": "Degraded" } From Hosted Control Plane Pod ingress-operator access https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg failed $ oc exec -it -n clusters-jwang-hcp-demo ingress-operator-6cb6866f5b-2b2h5 -- curl -v https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, availability-prober (init) * Uses proxy env variable NO_PROXY == 'kube-apiserver' * Uses proxy env variable HTTPS_PROXY == 'http://127.0.0.1:8090' * Trying 127.0.0.1:8090... * Connected to 127.0.0.1 (127.0.0.1) port 8090 (#0) * allocate connect buffer! * Establish HTTP proxy tunnel to canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443 > CONNECT canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443 HTTP/1.1 > Host: canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443 > User-Agent: curl/7.76.1 > Proxy-Connection: Keep-Alive > < HTTP/1.1 502 Bad Gateway < Content-Type: text/plain < Content-Length: 380 < * Received HTTP code 502 from proxy after CONNECT * CONNECT phase completed! * Closing connection 0 curl: (56) Received HTTP code 502 from proxy after CONNECT command terminated with exit code 56 Testing the Konnectivity server with curl $ sh /tmp/test-konnectivity.sh work directory is: /tmp/tmp.kI5bhZJWGA Forwarding from 127.0.0.1:8090 -> 8090 Forwarding from [::1]:8090 -> 8090 Handling connection for 8090 # HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated # TYPE aggregator_discovery_aggregation_count_total counter aggregator_discovery_aggregation_count_total 0 ...
Expected results:
Hostedcluster upgrade success
Additional info: