Description of problem:
CCO fails to check if the root credential has sufficient permissions for cr/cloud-credential-operator-gcp-ro-creds in passthrough mode.
Steps to Reproduce:
1. Create a GCP cluster with ClusterBot (4.14.0-0.nightly-2023-07-21-020713)
2. Switch to passthrough mode:
fxie@fxie-mac hive % oc edit cloudcredential
3. Remove the status subresource of cr/cloud-credential-operator-gcp-ro-creds to trigger a reconcile:
fxie@fxie-mac hive % oc edit credentialsrequest -n openshift-cloud-credential-operator cloud-credential-operator-gcp-ro-creds --subresource='status'
4. Check CR status:
fxie@fxie-mac hive % oc get credentialsrequest -n openshift-cloud-credential-operator cloud-credential-operator-gcp-ro-creds -o jsonpath='{.status}' | jq
{
"conditions": [
{
"lastProbeTime": "2023-07-21T10:30:29Z",
"lastTransitionTime": "2023-07-21T10:30:29Z",
"message": "failed to grant creds: error while validating permissions: error testing permissions: googleapi: Error 400: Permission advisorynotifications.notifications.get is not valid for this resource., badRequest",
"reason": "CredentialsProvisionFailure",
"status": "True",
"type": "CredentialsProvisionFailure"
}
],
"lastSyncGeneration": 0,
"provisioned": false
}
5. Check CCO status:
fxie@fxie-mac hive % oc get co cloud-credential -o jsonpath='{.status.conditions}' | jq
[
{
"lastTransitionTime": "2023-07-21T09:50:01Z",
"message": "All is well",
"reason": "AsExpected",
"status": "True",
"type": "Available"
},
{
"lastTransitionTime": "2023-07-21T10:30:29Z",
"message": "1 of 7 credentials requests are failing to sync.",
"reason": "CredentialsFailing",
"status": "True",
"type": "Degraded"
},
{
"lastTransitionTime": "2023-07-21T10:30:27Z",
"message": "6 of 7 credentials requests provisioned, 1 reporting errors.",
"reason": "Reconciling",
"status": "True",
"type": "Progressing"
},
{
"lastTransitionTime": "2023-07-21T09:50:01Z",
"message": "All is well",
"reason": "AsExpected",
"status": "True",
"type": "Upgradeable"
}
]
fxie@fxie-mac hive % oc logs -f cloud-credential-operator-6c5df89fc4-kjmj2 -n openshift-cloud-credential-operator -c cloud-credential-operator | grep -i "level=error"
time="2023-07-21T10:30:29Z" level=error msg="error syncing credentials: error while validating permissions: error testing permissions: googleapi: Error 400: Permission advisorynotifications.notifications.get is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2023-07-21T10:30:29Z" level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2023-07-21T10:30:31Z" level=error msg="error syncing credentials: error while validating permissions: error testing permissions: googleapi: Error 400: Permission advisorynotifications.notifications.get is not valid for this resource., badRequest" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
time="2023-07-21T10:30:31Z" level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds secret=openshift-cloud-credential-operator/cloud-credential-operator-gcp-ro-creds
...
Additional info:
We encountered the same problem on another cluster installed into the openshift-qe gcp project.
- is duplicated by
-
OCPBUGS-16808 CCO fails to check if the root credential has sufficient permissions for cr/cloud-credential-operator-gcp-ro-creds in passthrough mode
-
- Closed
-
- is related to
-
OCPBUGS-36140 GCP cluster with CCO Passthrough mode failed to install due to CCO degraded
-
- Closed
-