-
Bug
-
Resolution: Duplicate
-
Normal
-
None
-
4.12.z
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
No
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
CCO fails to check if the root credential has sufficient permissions for cr/cloud-credential-operator-gcp-ro-creds in passthrough mode
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. Install OCP 4.12.10 on GCP cluster.
2. Switch to passthrough mode:
$ oc get secret gcp-credentials -n kube-system -o yaml
~~~
<<snip>>
kind: Secret
metadata:
annotations:
cloudcredential.openshift.io/mode: passthrough
creationTimestamp: "2023-06-21T14:22:51Z"
name: gcp-credentials
namespace: kube-system
<<snip>>
~~~
3.
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
cloud-credential 4.12.10 True True True 25d
$ oc get co cloud-credential -o yaml
~~~
<<snip>>
- lastTransitionTime: "2023-07-15T19:10:27Z"
message: 1 of 7 credentials requests are failing to sync.
reason: CredentialsFailing
status: "True"
type: Degraded
- lastTransitionTime: "2023-07-15T19:10:27Z"
message: 6 of 7 credentials requests provisioned, 1 reporting errors.
reason: Reconciling
status: "True"
type: Progressing
~~~
4. cloud-credential-operator
~~~
apiVersion: cloudcredential.openshift.io/v1
kind: CredentialsRequest
<<snip>>
spec:
providerSpec:
apiVersion: cloudcredential.openshift.io/v1
kind: GCPProviderSpec
predefinedRoles:
- roles/iam.securityReviewer
- roles/iam.roleViewer
skipServiceCheck: true
secretRef:
name: cloud-credential-operator-gcp-ro-creds
namespace: openshift-cloud-credential-operator
serviceAccountNames:
- cloud-credential-operator
status:
conditions:
- lastProbeTime: "2023-07-15T19:10:27Z"
lastTransitionTime: "2023-07-15T19:10:27Z"
message: 'failed to grant creds: error determining whether a credentials update <--------
is needed'
reason: CredentialsProvisionFailure <------
status: "True" <------
type: CredentialsProvisionFailure 2.
~~~
5. Cloud-credential-operator pod logs:
~~~
$ oc logs pod/cloud-credential-operator-b5ff965b8-m2f4f -n openshift-cloud-credential-operator -c cloud-credential-operator
2023-07-17T10:27:56.479617663Z time="2023-07-17T10:27:56Z" level=error msg="error determining whether a credentials update is needed" actuator=gcp cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds error="error checking whether GCP client has sufficient permissions: error testing permissions: googleapi: Error 400: Permission advisorynotifications.notifications.get is not valid for this resource., badRequest"
2023-07-17T10:27:56.479691234Z time="2023-07-17T10:27:56Z" level=error msg="error syncing credentials: error determining whether a credentials update is needed" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
2023-07-17T10:27:56.479691234Z time="2023-07-17T10:27:56Z" level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
2023-07-17T10:27:58.480233923Z time="2023-07-17T10:27:58Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
~~~
After few days have passed, the issue has eventually self-resolved on all GCP clusters with no other actions taken.
Actual results:
The cloud-credential cluster operator is degraded due to credential requests failing to sync.
Expected results:
The cloud-credential cluster operator should be able to resync all credentialrequests without any issue.
Additional info:
Issue happened on all GCP cluster at the same time. Comparing this regex[1] with the error log from cloud-credential-operator[2], is it possible that the regex no longer matches what GCP API returns, and thus this permission isn't "filtered out" of the mentioned checks? [1] https://github.com/openshift/cloud-credential-operator/blob/e4ce607ad76b040422feec9625fcd0fb50b57d6b/pkg/operator/utils/gcp/utils.go#L86
- duplicates
-
OCPBUGS-16646 CCO fails to check if the root credential has sufficient permissions for cr/cloud-credential-operator-gcp-ro-creds in passthrough mode
-
- Closed
-