Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-43856

Cloud Credential Operator down due to GCP backupdr.googleapis.com access request

XMLWordPrintable

    • None
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, the Cloud Credential Operator (CCO) for a cluster installed on Google Cloud Platform (GCP) was degraded when a request was sent from backupdr.googleapis.com. With this release, the skipServiceCheck parameter is set to true on the CredentialsRequest custom resource (CR), so this issue no longer occurs.
      ====

      Cause – GCP upstream change
      Consequence – CCO becomes degraded
      Fix – Update exposed CredentialsRequests to set skipServiceCheck
      Result – CCO is not degraded anymore
      Show
      Previously, the Cloud Credential Operator (CCO) for a cluster installed on Google Cloud Platform (GCP) was degraded when a request was sent from backupdr.googleapis.com. With this release, the skipServiceCheck parameter is set to true on the CredentialsRequest custom resource (CR), so this issue no longer occurs. ==== Cause – GCP upstream change Consequence – CCO becomes degraded Fix – Update exposed CredentialsRequests to set skipServiceCheck Result – CCO is not degraded anymore
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-43821. The following is the description of the original issue:
      —
      Description of problem:

          Cloud Credential Operator is down in GCP cluster. Cluster is trying to access API endpoint backupdr.googleapis.com even though this API is not under customer's subscription.


CCO has these logs:
time="2024-10-25T01:12:30Z" level=warning msg="Detected required APIs that are disabled: [backupdr.googleapis.com]" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2024-10-25T01:12:30Z" level=error msg="not all required service APIs are enabled" actuator=gcp cr=openshift-cloud-credential-operator/openshift-machine-api-gcp
time="2024-10-25T01:12:30Z" level=error msg="error syncing credentials: not all required service APIs are enabled" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-gcp secret=openshift-machine-api/gcp-cloud-credentials
time="2024-10-25T01:12:30Z" level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/openshift-machine-api-gcp secret=openshift-machine-api/gcp-cloud-credentials  

      Version-Release number of selected component (if applicable):

      How reproducible:

      Not determinable yet. We have experienced this for a specific cluster. We still don't know where the request is originating.

      Steps to Reproduce:

          1.
    2.
    3.

      Actual results:

          Cloud Credential Operator is down because cluster is trying to access an endpoint that the cluster is not subscribed to.

      Expected results:

          Cluster should not access endpoints it is not subscribed to.

      Additional info:

      OHSS: https://issues.redhat.com/browse/OHSS-38255 

      Discussion thread: https://redhat-internal.slack.com/archives/CCX9DB894/p1729820675700629 

              jstuever@redhat.com Jeremiah Stuever
              openshift-crt-jira-prow OpenShift Prow Bot
              Jianping Shu Jianping Shu
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: