Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28787

CCO degrade when remove root credential for GCP cluster in Mint mode

XMLWordPrintable

    • Yes
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, removing the root credential from a {gcp-first} cluster that used the Cloud Credential Operator (CCO) in mint mode caused the CCO to become degraded after approximately one hour.
      In a degraded state, the CCO cannot manage the component credential secrets on a cluster.
      The issue is resolved in this release.
      (link:https://issues.redhat.com/browse/OCPBUGS-28787[*OCPBUGS-28787*])
      Show
      * Previously, removing the root credential from a {gcp-first} cluster that used the Cloud Credential Operator (CCO) in mint mode caused the CCO to become degraded after approximately one hour. In a degraded state, the CCO cannot manage the component credential secrets on a cluster. The issue is resolved in this release. (link: https://issues.redhat.com/browse/OCPBUGS-28787 [* OCPBUGS-28787 *])
    • Bug Fix
    • Done

      Description of problem:

      It was found when testing OCP-71263 and regression OCP-35770 for 4.15.
      For GCP in Mint mode, the root credential can be removed after cluster installation.
      But after removing the root credential, CCO became degrade.      

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2024-01-25-051548
      
      4.15.0-rc.3

      How reproducible:

          
      Always    

      Steps to Reproduce:

          1.Install a GCP cluster with Mint mode
      
          2.After install, remove the root credential
      jianpingshu@jshu-mac ~ % oc delete secret -n kube-system gcp-credentials
      secret "gcp-credentials" deleted     
      
          3.Wait some time(about 1/2h to 1h), CCO became degrade 
          
      jianpingshu@jshu-mac ~ % oc get co cloud-credential
      NAME               VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      cloud-credential   4.15.0-rc.3   True        True          True       6h45m   6 of 7 credentials requests are failing to sync.
      
      jianpingshu@jshu-mac ~ % oc -n openshift-cloud-credential-operator get -o json credentialsrequests | jq -r '.items[] | select(tostring | contains("InfrastructureMismatch") | not) | .metadata.name as $n | .status.conditions // [{type: "NoConditions"}] | .[] | .type + "=" + .status + " " + $n + " " + .reason + ": " + .message' | sort
      CredentialsProvisionFailure=False openshift-cloud-network-config-controller-gcp CredentialsProvisionSuccess: successfully granted credentials request
      CredentialsProvisionFailure=True cloud-credential-operator-gcp-ro-creds CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-gcp-ccm CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-gcp-pd-csi-driver-operator CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-image-registry-gcs CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-ingress-gcp CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-machine-api-gcp CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      
      openshift-cloud-network-config-controller-gcp has no failure because it doesn't has customized role in 4.15.0.rc3

      Actual results:

       CCO became degrade

      Expected results:

       CCO not in degrade, just "upgradeable" condition updated with missing the root credential

      Additional info:

      Tested the same case on 4.14.10, no issue 

       

              jstuever@redhat.com Jeremiah Stuever
              jshu@redhat.com Jianping Shu
              Jianping Shu Jianping Shu
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: