Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-28787

CCO degrade when remove root credential for GCP cluster in Mint mode

    XMLWordPrintable

Details

    • Yes
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      *Cause*: when the root credential was removed while the GCP cluster was in mint mode, the cloud credential operator would become degraded after approximately an hour.
      *Consequence*: cloud credential operator would become degraded and can’t manage the component credential secrets.
      *Fix*: The code was enhanced to consider the custom role and avoid unnecessary credential update. 
      *Result*: when the root credential was removed while the GCP cluster was in mint mode, the cloud credential operator would not become degraded any more.
      Show
      *Cause*: when the root credential was removed while the GCP cluster was in mint mode, the cloud credential operator would become degraded after approximately an hour. *Consequence*: cloud credential operator would become degraded and can’t manage the component credential secrets. *Fix*: The code was enhanced to consider the custom role and avoid unnecessary credential update.  *Result*: when the root credential was removed while the GCP cluster was in mint mode, the cloud credential operator would not become degraded any more.
    • Bug Fix
    • In Progress

    Description

      Description of problem:

      It was found when testing OCP-71263 and regression OCP-35770 for 4.15.
      For GCP in Mint mode, the root credential can be removed after cluster installation.
      But after removing the root credential, CCO became degrade.      

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2024-01-25-051548
      
      4.15.0-rc.3

      How reproducible:

          
      Always    

      Steps to Reproduce:

          1.Install a GCP cluster with Mint mode
      
          2.After install, remove the root credential
      jianpingshu@jshu-mac ~ % oc delete secret -n kube-system gcp-credentials
      secret "gcp-credentials" deleted     
      
          3.Wait some time(about 1/2h to 1h), CCO became degrade 
          
      jianpingshu@jshu-mac ~ % oc get co cloud-credential
      NAME               VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      cloud-credential   4.15.0-rc.3   True        True          True       6h45m   6 of 7 credentials requests are failing to sync.
      
      jianpingshu@jshu-mac ~ % oc -n openshift-cloud-credential-operator get -o json credentialsrequests | jq -r '.items[] | select(tostring | contains("InfrastructureMismatch") | not) | .metadata.name as $n | .status.conditions // [{type: "NoConditions"}] | .[] | .type + "=" + .status + " " + $n + " " + .reason + ": " + .message' | sort
      CredentialsProvisionFailure=False openshift-cloud-network-config-controller-gcp CredentialsProvisionSuccess: successfully granted credentials request
      CredentialsProvisionFailure=True cloud-credential-operator-gcp-ro-creds CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-gcp-ccm CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-gcp-pd-csi-driver-operator CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-image-registry-gcs CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-ingress-gcp CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      CredentialsProvisionFailure=True openshift-machine-api-gcp CredentialsProvisionFailure: failed to grant creds: unable to fetch root cloud cred secret: Secret "gcp-credentials" not found
      
      openshift-cloud-network-config-controller-gcp has no failure because it doesn't has customized role in 4.15.0.rc3

      Actual results:

       CCO became degrade

      Expected results:

       CCO not in degrade, just "upgradeable" condition updated with missing the root credential

      Additional info:

      Tested the same case on 4.14.10, no issue 

       

      Attachments

        Issue Links

          Activity

            People

              jstuever@redhat.com Jeremiah Stuever
              jshu@redhat.com Jianping Shu
              Jianping Shu Jianping Shu
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: