Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-57262

disconnected hostedcluster upgrade failed and report CanaryChecksRepetitiveFailures

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.16, 4.17
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      disconnected hostedcluster upgrade failed(4.16.17 to 4.17.5) and report CanaryChecksRepetitiveFailures

      Version-Release number of selected component (if applicable):

      OCP Version: 4.18.10
      MCE: 2.8.2
      OpenShift Virtualization: 4.18.2
      
      guest cluster
      ---
      OCP Version: 4.16.17

      How reproducible:

      upgrade hostedcluster from 4.16.17 to 4.17.5    

      Steps to Reproduce:

      1. 
      oc get HostedCluster jwang-hcp-demo -o json | jq -r '.spec.release.image="helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64"' | oc apply -f -     
      
      2.
      oc annotate hostedcluster -n clusters jwang-hcp-demo "hypershift.openshift.io/force-upgrade-to=helper.ocp.ap.vwg:5000/ocp4/openshift4:4.17.5-x86_64" --overwrite  

      Actual results:

      $ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress 
      NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      ingress   4.17.5    True        False         True       19h     The "default" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:...
      
      $ oc --kubeconfig=/root/jwang-hcp-demo-kubeconfig get co ingress -o json | jq .status.conditions[2] 
      {
        "lastTransitionTime": "2025-06-04T03:10:42Z",
        "message": "The \"default\" ingress controller reports Degraded=True: DegradedConditions: One or more other status conditions indicate a degraded state: CanaryChecksSucceeding=False (CanaryChecksRepetitiveFailures: Canary route checks for the default ingress controller are failing. Last 1 error messages:\nerror sending canary HTTP request to \"canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Get \"https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg\": Bad Gateway (x1112 over 18h31m57s))",
        "reason": "IngressDegraded",
        "status": "True",
        "type": "Degraded"
      }
      
      From Hosted Control Plane Pod ingress-operator access https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg failed
      $ oc exec -it -n clusters-jwang-hcp-demo ingress-operator-6cb6866f5b-2b2h5 -- curl -v https://canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg 
      Defaulted container "ingress-operator" out of: ingress-operator, konnectivity-proxy, availability-prober (init)
      * Uses proxy env variable NO_PROXY == 'kube-apiserver'
      * Uses proxy env variable HTTPS_PROXY == 'http://127.0.0.1:8090'
      *   Trying 127.0.0.1:8090...
      * Connected to 127.0.0.1 (127.0.0.1) port 8090 (#0)
      * allocate connect buffer!
      * Establish HTTP proxy tunnel to canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443
      > CONNECT canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443 HTTP/1.1
      > Host: canary-openshift-ingress-canary.apps.jwang-hcp-demo.ap.vwg:443
      > User-Agent: curl/7.76.1
      > Proxy-Connection: Keep-Alive
      > 
      < HTTP/1.1 502 Bad Gateway
      < Content-Type: text/plain
      < Content-Length: 380
      < 
      * Received HTTP code 502 from proxy after CONNECT
      * CONNECT phase completed!
      * Closing connection 0
      curl: (56) Received HTTP code 502 from proxy after CONNECT
      command terminated with exit code 56
      
      Testing the Konnectivity server with curl
      $ sh /tmp/test-konnectivity.sh
      work directory is: /tmp/tmp.kI5bhZJWGA
      Forwarding from 127.0.0.1:8090 -> 8090
      Forwarding from [::1]:8090 -> 8090
      Handling connection for 8090
      # HELP aggregator_discovery_aggregation_count_total [ALPHA] Counter of number of times discovery was aggregated
      # TYPE aggregator_discovery_aggregation_count_total counter
      aggregator_discovery_aggregation_count_total 0
      ...    

      Expected results:

      Hostedcluster upgrade success    

      Additional info:

          

              ocohen@redhat.com Oren Cohen
              wang.jun.1974 Jun Wang
              None
              None
              Yu Li Yu Li
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: