Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36140

GCP cluster with CCO Passthrough mode failed to install due to CCO degraded

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • None
    • 4.13.z, 4.14.z
    • Cloud Credential Operator
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, when checking whether passthrough permissions are sufficient, the Cloud Credential Operator sometimes received a response from the Google Cloud Platform API that a permission is invalid for a project.
      This response caused the Operator to become degraded and installation to fail.
      With this release, the Operator is updated to handle this error gracefully.
      (link:https://issues.redhat.com/browse/OCPBUGS-36140[*OCPBUGS-36140*])
      Show
      * Previously, when checking whether passthrough permissions are sufficient, the Cloud Credential Operator sometimes received a response from the Google Cloud Platform API that a permission is invalid for a project. This response caused the Operator to become degraded and installation to fail. With this release, the Operator is updated to handle this error gracefully. (link: https://issues.redhat.com/browse/OCPBUGS-36140 [* OCPBUGS-36140 *])
    • Bug Fix
    • Done

      Description of problem:

      GCP private cluster with CCO Passthrough mode failed to install due to CCO degraded.
status:  
conditions:  - lastTransitionTime: "2024-06-24T06:04:39Z"    message: 1 of 7 credentials requests are failing to sync.    reason: CredentialsFailing    status: "True"    type: Degraded    

      Version-Release number of selected component (if applicable):

      4.13.0-0.nightly-2024-06-21-203120    

      How reproducible:

      Always    

      Steps to Reproduce:

          1.Create GCP private cluster with CCO Passthrough mode, flexy template is private-templates/functionality-testing/aos-4_13/ipi-on-gcp/versioned-installer-xpn-private     
    2.Wait for cluster installation

      Actual results:

      jianpingshu@jshu-mac ~ % oc get clusterversionNAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUSversion             False       False         23m     Error while reconciling 4.13.0-0.nightly-2024-06-21-203120: the cluster operator cloud-credential is degraded

status:  
conditions:  - lastTransitionTime: "2024-06-24T06:04:39Z"    message: 1 of 7 credentials requests are failing to sync.    reason: CredentialsFailing    status: "True"    type: Degraded

jianpingshu@jshu-mac ~ % oc -n openshift-cloud-credential-operator get -o json credentialsrequests | jq -r '.items[] | select(tostring | contains("InfrastructureMismatch") | not) | .metadata.name as $n | .status.conditions // [{type: "NoConditions"}] | .[] | .type + "=" + .status + " " + $n + " " + .reason + ": " + .message' | sortCredentialsProvisionFailure=True cloud-credential-operator-gcp-ro-creds CredentialsProvisionFailure: failed to grant creds: error while validating permissions: error testing permissions: googleapi: Error 400: Permission commerceoffercatalog.agreements.list is not valid for this resource., badRequest
NoConditions= openshift-cloud-network-config-controller-gcp :
NoConditions= openshift-gcp-ccm :
NoConditions= openshift-gcp-pd-csi-driver-operator :
NoConditions= openshift-image-registry-gcs :
NoConditions= openshift-ingress-gcp :
NoConditions= openshift-machine-api-gcp :    

      Expected results:

      Cluster installed successfully without degrade    

      Additional info:

      Some problem PROW CI tests: 
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.14-multi-nightly-gcp-ipi-user-labels-tags-filestore-csi-tp-arm-f14/1805064266043101184
https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.14-amd64-nightly-4.14-upgrade-from-stable-4.13-gcp-ipi-xpn-fips-f28/1804676149503070208    

       

              jstuever@redhat.com Jeremiah Stuever
              jshu@redhat.com Jianping Shu
              Jianping Shu Jianping Shu
              Jianli Wei
              Votes:
              2 Vote for this issue
              Watchers:
              17 Start watching this issue

                Created:
                Updated:
                Resolved: