Image registry QE is seeing an error on GCP OIDC fips jobs in relation to cert rotation (Job sample link below).

We're seeing the following error when the operator attempts to communicate with GCS:

API:E0313 17:13:27.917931       1 controller.go:377] unable to sync: unable to sync storage configuration: Get "https://storage.googleapis.com/storage/v1/b/ci-op-x562pvjc-61c45-mptnw-image-registry-us-central1-pnwmaiue?alt=json&prettyPrint=false&projection=full": oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/ci-op-x562pv-openshift-i-946z7@openshift-qe.iam.gserviceaccount.com:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}, requeuingemphasis on oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}.

When looking at the CCO pod logs, we observe:

time="2024-03-13T16:15:01Z" level=error msg="error creating GCP client" error="Secret \"gcp-credentials\" not found"
time="2024-03-13T16:15:01Z" level=error msg="error determining whether a credentials update is needed" actuator=gcp cr=openshift-cloud-credential-operator/openshift-image-registry-gcs error="unable to check whether credentialsRequest needs update"
time="2024-03-13T16:15:01Z" level=error msg="error syncing credentials: error determining whether a credentials update is needed" controller=credreq cr=openshift-cloud-credential-operator/openshift-image-registry-gcs secret=openshift-image-registry/installer-cloud-credentials

The job artifacts for reference: https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-o[…]n-cert-f14/gather-extra/artifacts/pods/

    4.16, 4.15, 4.14

    Not sure (Wen please help)

    1.
    2.
    3.
    

    Image registry cannot communicate with cloud storage

    Image registry can communicate with cloud storage