Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23919

CCO is unable to mint credentials after enabling the CloudCredential capability in day 2

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.15
    • Cloud Credential Operator
    • None
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Users have to either label all target Secrets with cloudcredential.openshift.io/credentials-request: true or delete all target Secrets so that they are minted.

      Steps to Reproduce:

      1. Create a cluster with CCO disabled on AWS, enable the CloudCredential cap in day 2, see https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-68220 
2. co/cloud-credential reports that the credentials requests are failing to sync

      CCO logs:

      time="2023-11-24T09:59:28Z" level=info msg="syncing credentials request" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="found secret namespace" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="running sync" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="running Exists" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="target secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="stsDetected: false" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="provisioning with cred minting" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="Loading infrastructure name: fxie-manual-1-sh87h" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="loading AWS credentials from secret" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=kube-system/aws-creds
time="2023-11-24T09:59:28Z" level=debug msg="creating root AWS client" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=kube-system/aws-creds
time="2023-11-24T09:59:28Z" level=debug msg="loading AWS credentials from secret" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2023-11-24T09:59:28Z" level=warning msg="read-only creds not found, using root creds client" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-cloud-credential-operator/cloud-credential-operator-iam-ro-creds
time="2023-11-24T09:59:28Z" level=debug msg="loading AWS credentials from secret" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=kube-system/aws-creds
time="2023-11-24T09:59:28Z" level=debug msg="creating root AWS client" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress secret=kube-system/aws-creds
time="2023-11-24T09:59:28Z" level=info msg="user exists" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress userName=fxie-manual-1-sh87h-openshift-ingress-fb8z6
time="2023-11-24T09:59:28Z" level=debug msg="loading cluster version to read clusterID" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="found cluster ID" actuator=aws clusterID=9c352a9e-3879-49e0-88fb-cff20e872df0 cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="desired user policy: {\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"elasticloadbalancing:DescribeLoadBalancers\",\"route53:ListHostedZones\",\"route53:ListTagsForResources\",\"route53:ChangeResourceRecordSets\",\"tag:GetResources\",\"sts:AssumeRole\"],\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":[\"iam:GetUser\"],\"Resource\":\"arn:aws:iam::301721915996:user/fxie-manual-1-sh87h-openshift-ingress-fb8z6\"}]}" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="current user policy: {\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Action\":[\"elasticloadbalancing:DescribeLoadBalancers\",\"route53:ListHostedZones\",\"route53:ListTagsForResources\",\"route53:ChangeResourceRecordSets\",\"tag:GetResources\",\"sts:AssumeRole\"],\"Resource\":\"*\"},{\"Effect\":\"Allow\",\"Action\":[\"iam:GetUser\"],\"Resource\":\"arn:aws:iam::301721915996:user/fxie-manual-1-sh87h-openshift-ingress-fb8z6\"}]}" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="no changes to user policy" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="sync ListAccessKeys" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="secret does not exist" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=debug msg="access key exists? false" accessKeyID= actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="generating new AWS access key" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="deleting all AWS access keys" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="deleting access key" accessKeyID=... actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="all access keys deleted" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="access key created" accessKeyID=... actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress
time="2023-11-24T09:59:28Z" level=info msg="processing secret" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress targetSecret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=info msg="processed secret" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress operation=unchanged targetSecret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=error msg="error syncing creds in mint-mode" actuator=aws cr=openshift-cloud-credential-operator/openshift-ingress error="error processing secret"
time="2023-11-24T09:59:28Z" level=error msg="error syncing credentials: error syncing creds in mint-mode: error processing secret" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=error msg="errored with condition: CredentialsProvisionFailure" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=debug msg="updating credentials request status" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-ingress-operator/cloud-credentials
time="2023-11-24T09:59:28Z" level=debug msg="status unchanged" controller=credreq cr=openshift-cloud-credential-operator/openshift-ingress secret=openshift-ingress-operator/cloud-credentials

      Additional info:

      This function errors out after the transition to mint mode. This is due to:

      • The manually created credentials do not present in the controller-runtime client's cache since they do not fulfill the labelSelectors.
      • When calling CreateOrPatch on each of these Secrets, the controller-runtime client GETs the Secret from its cache, which fails, tries to CREATE the Secret, which fails again as the Secret actually exists (just not in the cache). 

            Unassigned Unassigned
            fxierh Feilian Xie
            Feilian Xie Feilian Xie
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: